Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add miscellaneous block to handle structure trace overflow #9733

Merged
merged 29 commits into from
Nov 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
653ed96
basic mega honk tests passes with block overflow
ledwards2225 Nov 4, 2024
6b96884
Merge branch 'master' into lde/misc_block
ledwards2225 Nov 4, 2024
07c8f0c
reinstate lookup/databus constants
ledwards2225 Nov 5, 2024
172aef0
basic tests for aux passing w caveats
ledwards2225 Nov 5, 2024
8cfbb63
add missing miscellaneous objects
ledwards2225 Nov 5, 2024
dc2a627
correctly updating mem records
ledwards2225 Nov 5, 2024
8b7e955
fix ultra arith
ledwards2225 Nov 6, 2024
b73c7fb
fix missing selector ref; aux tests passes at arbitrary split point
ledwards2225 Nov 6, 2024
edfe90c
WiP introduce trace setting concept
ledwards2225 Nov 6, 2024
97d4ad6
everything building with trace settings
ledwards2225 Nov 7, 2024
354761c
basic IVC overflow test passes
ledwards2225 Nov 7, 2024
3dbcd5a
remove circuit size pverride mechanism
ledwards2225 Nov 7, 2024
bbecf8f
Merge branch 'master' into lde/misc_block
ledwards2225 Nov 7, 2024
8b81549
naming change to overflow
ledwards2225 Nov 7, 2024
db06461
cleanup pass
ledwards2225 Nov 7, 2024
29e5ce2
more clean and reorg
ledwards2225 Nov 7, 2024
6e2a49f
add megaHonk test for various types of overflow
ledwards2225 Nov 7, 2024
ea81b18
update IVC test
ledwards2225 Nov 7, 2024
9e35acf
comments
ledwards2225 Nov 7, 2024
02ba4bb
test name update
ledwards2225 Nov 7, 2024
b014c9e
comments and clean
ledwards2225 Nov 7, 2024
93bb09d
comment
ledwards2225 Nov 8, 2024
5e65b9f
Merge branch 'master' into lde/misc_block
ledwards2225 Nov 8, 2024
aedac34
only run overflow test w mega
ledwards2225 Nov 8, 2024
24eaaaf
update join split circuit hash based on addition of new block
ledwards2225 Nov 8, 2024
6776c4c
actually disable new mega test for zk flavor
ledwards2225 Nov 8, 2024
21dc52e
fix StructuredTraceOverflow test for MegaZKFlavor
iakovenkos Nov 8, 2024
94a11b1
cleanup and clarify comments per review
ledwards2225 Nov 11, 2024
4b12b1f
Merge branch 'lde/misc_block' of github.com:AztecProtocol/aztec-packa…
ledwards2225 Nov 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions barretenberg/cpp/src/barretenberg/bb/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -359,7 +359,7 @@ void client_ivc_prove_output_all_msgpack(const std::string& bytecodePath,
// TODO(https://github.com/AztecProtocol/barretenberg/issues/1101): remove use of auto_verify_mode
ClientIVC ivc;
ivc.auto_verify_mode = true;
ivc.trace_structure = TraceStructure::E2E_FULL_TEST;
ivc.trace_settings.structure = TraceStructure::E2E_FULL_TEST;

// Accumulate the entire program stack into the IVC
// TODO(https://github.com/AztecProtocol/barretenberg/issues/1116): remove manual setting of is_kernel once databus
Expand Down Expand Up @@ -448,7 +448,7 @@ bool foldAndVerifyProgram(const std::string& bytecodePath, const std::string& wi

ClientIVC ivc;
ivc.auto_verify_mode = true;
ivc.trace_structure = TraceStructure::SMALL_TEST;
ivc.trace_settings.structure = TraceStructure::SMALL_TEST;

auto program_stack = acir_format::get_acir_program_stack(
bytecodePath, witnessPath, false); // TODO(https://github.com/AztecProtocol/barretenberg/issues/1013): this
Expand Down Expand Up @@ -501,7 +501,7 @@ void client_ivc_prove_output_all(const std::string& bytecodePath,
// TODO(https://github.com/AztecProtocol/barretenberg/issues/1101): remove use of auto_verify_mode
ClientIVC ivc;
ivc.auto_verify_mode = true;
ivc.trace_structure = TraceStructure::E2E_FULL_TEST;
ivc.trace_settings.structure = TraceStructure::E2E_FULL_TEST;

auto program_stack = acir_format::get_acir_program_stack(
bytecodePath, witnessPath, false); // TODO(https://github.com/AztecProtocol/barretenberg/issues/1013): this
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ class ClientIVCBench : public benchmark::Fixture {
BENCHMARK_DEFINE_F(ClientIVCBench, Full)(benchmark::State& state)
{
ClientIVC ivc;
ivc.trace_structure = TraceStructure::CLIENT_IVC_BENCH;
ivc.trace_settings.structure = TraceStructure::CLIENT_IVC_BENCH;
auto total_num_circuits = 2 * static_cast<size_t>(state.range(0)); // 2x accounts for kernel circuits
auto mocked_vkeys = mock_verification_keys(total_num_circuits);

Expand Down
10 changes: 5 additions & 5 deletions barretenberg/cpp/src/barretenberg/client_ivc/client_ivc.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -173,11 +173,11 @@ void ClientIVC::accumulate(ClientCircuit& circuit, const std::shared_ptr<Verific
// Construct the proving key for circuit
std::shared_ptr<DeciderProvingKey> proving_key;
if (!initialized) {
proving_key = std::make_shared<DeciderProvingKey>(circuit, trace_structure);
trace_usage_tracker = ExecutionTraceUsageTracker(trace_structure);
proving_key = std::make_shared<DeciderProvingKey>(circuit, trace_settings);
trace_usage_tracker = ExecutionTraceUsageTracker(trace_settings);
} else {
proving_key = std::make_shared<DeciderProvingKey>(
circuit, trace_structure, fold_output.accumulator->proving_key.commitment_key);
circuit, trace_settings, fold_output.accumulator->proving_key.commitment_key);
}

// Update the accumulator trace usage based on the present circuit
Expand Down Expand Up @@ -370,10 +370,10 @@ std::vector<std::shared_ptr<ClientIVC::VerificationKey>> ClientIVC::precompute_f
}

// Reset the scheme so it can be reused for actual accumulation, maintaining the trace structure setting as is
TraceStructure structure = trace_structure;
TraceSettings settings = trace_settings;
bool auto_verify = auto_verify_mode;
*this = ClientIVC();
this->trace_structure = structure;
this->trace_settings = settings;
this->auto_verify_mode = auto_verify;

return vkeys;
Expand Down
4 changes: 2 additions & 2 deletions barretenberg/cpp/src/barretenberg/client_ivc/client_ivc.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,8 @@ class ClientIVC {
// Management of linking databus commitments between circuits in the IVC
DataBusDepot bus_depot;

// A flag indicating whether or not to construct a structured trace in the DeciderProvingKey
TraceStructure trace_structure = TraceStructure::NONE;
// Settings related to the use of fixed block sizes for each gate in the execution trace
TraceSettings trace_settings;

// TODO(https://github.com/AztecProtocol/barretenberg/issues/1101): eventually do away with this.
// Setting auto_verify_mode = true will cause kernel completion logic to be added to kernels automatically
Expand Down
57 changes: 43 additions & 14 deletions barretenberg/cpp/src/barretenberg/client_ivc/client_ivc.test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -76,11 +76,11 @@ class ClientIVCTests : public ::testing::Test {
}

auto precompute_verification_keys(const size_t num_circuits,
TraceStructure trace_structure,
TraceSettings trace_settings,
size_t log2_num_gates = 16)
{
ClientIVC ivc; // temporary IVC instance needed to produce the complete kernel circuits
ivc.trace_structure = trace_structure;
ivc.trace_settings = trace_settings;

std::vector<std::shared_ptr<VerificationKey>> vkeys;

Expand Down Expand Up @@ -163,7 +163,7 @@ TEST_F(ClientIVCTests, BadProofFailure)
// Confirm that the IVC verifies if nothing is tampered with
{
ClientIVC ivc;
ivc.trace_structure = TraceStructure::SMALL_TEST;
ivc.trace_settings.structure = TraceStructure::SMALL_TEST;

MockCircuitProducer circuit_producer;

Expand All @@ -179,7 +179,7 @@ TEST_F(ClientIVCTests, BadProofFailure)
// The IVC throws an exception if the FIRST fold proof is tampered with
{
ClientIVC ivc;
ivc.trace_structure = TraceStructure::SMALL_TEST;
ivc.trace_settings.structure = TraceStructure::SMALL_TEST;

MockCircuitProducer circuit_producer;

Expand All @@ -204,7 +204,7 @@ TEST_F(ClientIVCTests, BadProofFailure)
// The IVC fails if the SECOND fold proof is tampered with
{
ClientIVC ivc;
ivc.trace_structure = TraceStructure::SMALL_TEST;
ivc.trace_settings.structure = TraceStructure::SMALL_TEST;

MockCircuitProducer circuit_producer;

Expand All @@ -229,7 +229,7 @@ TEST_F(ClientIVCTests, BadProofFailure)
// The IVC fails if the 3rd/FINAL fold proof is tampered with
{
ClientIVC ivc;
ivc.trace_structure = TraceStructure::SMALL_TEST;
ivc.trace_settings.structure = TraceStructure::SMALL_TEST;

MockCircuitProducer circuit_producer;

Expand Down Expand Up @@ -278,7 +278,7 @@ TEST_F(ClientIVCTests, BasicLarge)
TEST_F(ClientIVCTests, BasicStructured)
{
ClientIVC ivc;
ivc.trace_structure = TraceStructure::SMALL_TEST;
ivc.trace_settings.structure = TraceStructure::SMALL_TEST;

MockCircuitProducer circuit_producer;

Expand Down Expand Up @@ -307,7 +307,7 @@ TEST_F(ClientIVCTests, PrecomputedVerificationKeys)

MockCircuitProducer circuit_producer;

auto precomputed_vks = circuit_producer.precompute_verification_keys(NUM_CIRCUITS, TraceStructure::NONE);
auto precomputed_vks = circuit_producer.precompute_verification_keys(NUM_CIRCUITS, TraceSettings{});

// Construct and accumulate set of circuits using the precomputed vkeys
for (size_t idx = 0; idx < NUM_CIRCUITS; ++idx) {
Expand All @@ -325,15 +325,15 @@ TEST_F(ClientIVCTests, PrecomputedVerificationKeys)
TEST_F(ClientIVCTests, StructuredPrecomputedVKs)
{
ClientIVC ivc;
ivc.trace_structure = TraceStructure::SMALL_TEST;
ivc.trace_settings.structure = TraceStructure::SMALL_TEST;

size_t NUM_CIRCUITS = 4;
size_t log2_num_gates = 5; // number of gates in baseline mocked circuit

MockCircuitProducer circuit_producer;

auto precomputed_vks =
circuit_producer.precompute_verification_keys(NUM_CIRCUITS, ivc.trace_structure, log2_num_gates);
circuit_producer.precompute_verification_keys(NUM_CIRCUITS, ivc.trace_settings, log2_num_gates);

// Construct and accumulate set of circuits using the precomputed vkeys
for (size_t idx = 0; idx < NUM_CIRCUITS; ++idx) {
Expand All @@ -357,10 +357,10 @@ TEST(ClientIVCBenchValidation, Full6)
bb::srs::init_grumpkin_crs_factory("../srs_db/grumpkin");

ClientIVC ivc;
ivc.trace_structure = TraceStructure::CLIENT_IVC_BENCH;
ivc.trace_settings.structure = TraceStructure::CLIENT_IVC_BENCH;
size_t total_num_circuits{ 12 };
PrivateFunctionExecutionMockCircuitProducer circuit_producer;
auto precomputed_vkeys = circuit_producer.precompute_verification_keys(total_num_circuits, ivc.trace_structure);
auto precomputed_vkeys = circuit_producer.precompute_verification_keys(total_num_circuits, ivc.trace_settings);
perform_ivc_accumulation_rounds(total_num_circuits, ivc, precomputed_vkeys);
auto proof = ivc.prove();
bool verified = verify_ivc(proof, ivc);
Expand All @@ -377,7 +377,7 @@ TEST(ClientIVCBenchValidation, Full6MockedVKs)
bb::srs::init_grumpkin_crs_factory("../srs_db/grumpkin");

ClientIVC ivc;
ivc.trace_structure = TraceStructure::CLIENT_IVC_BENCH;
ivc.trace_settings.structure = TraceStructure::CLIENT_IVC_BENCH;
size_t total_num_circuits{ 12 };
PrivateFunctionExecutionMockCircuitProducer circuit_producer;
auto mocked_vkeys = mock_verification_keys(total_num_circuits);
Expand All @@ -386,4 +386,33 @@ TEST(ClientIVCBenchValidation, Full6MockedVKs)
verify_ivc(proof, ivc);
};
ASSERT_NO_FATAL_FAILURE(run_test());
}
}

/**
* @brief Test use of structured trace overflow block mechanism
* @details Accumulate 4 circuits which have progressively more arithmetic gates. The final two overflow the prescribed
* arithmetic block size and make use of the overflow block which has sufficient capacity.
*
*/
TEST_F(ClientIVCTests, StructuredTraceOverflow)
{
ClientIVC ivc;

// Define trace settings with sufficient overflow capacity to accommodate each of the circuits to be accumulated
uint32_t overflow_capacity = 1 << 17;
ivc.trace_settings = { TraceStructure::SMALL_TEST, overflow_capacity };

MockCircuitProducer circuit_producer;

size_t NUM_CIRCUITS = 4;

// Construct and accumulate some circuits of varying size
size_t log2_num_gates = 14;
for (size_t idx = 0; idx < NUM_CIRCUITS; ++idx) {
auto circuit = circuit_producer.create_next_circuit(ivc, log2_num_gates);
ivc.accumulate(circuit);
log2_num_gates += 1;
}

EXPECT_TRUE(ivc.prove_and_verify());
};
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ TEST_F(ClientIVCAutoVerifyTests, BasicStructured)
{
ClientIVC ivc;
ivc.auto_verify_mode = true;
ivc.trace_structure = TraceStructure::SMALL_TEST;
ivc.trace_settings.structure = TraceStructure::SMALL_TEST;

// Construct some circuits of varying size
Builder circuit_0 = create_mock_circuit(ivc, /*is_kernel=*/false, /*log2_num_gates=*/5);
Expand Down Expand Up @@ -185,7 +185,7 @@ TEST_F(ClientIVCAutoVerifyTests, StructuredPrecomputedVKs)
{
ClientIVC ivc;
ivc.auto_verify_mode = true;
ivc.trace_structure = TraceStructure::SMALL_TEST;
ivc.trace_settings.structure = TraceStructure::SMALL_TEST;

// Construct a set of arbitrary circuits
size_t NUM_CIRCUITS = 4;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ class ClientIVCIntegrationTests : public ::testing::Test {
TEST_F(ClientIVCIntegrationTests, BenchmarkCaseSimple)
{
ClientIVC ivc;
ivc.trace_structure = TraceStructure::CLIENT_IVC_BENCH;
ivc.trace_settings.structure = TraceStructure::CLIENT_IVC_BENCH;

MockCircuitProducer circuit_producer;

Expand All @@ -61,7 +61,7 @@ TEST_F(ClientIVCIntegrationTests, BenchmarkCaseSimple)
TEST_F(ClientIVCIntegrationTests, ConsecutiveKernels)
{
ClientIVC ivc;
ivc.trace_structure = TraceStructure::CLIENT_IVC_BENCH;
ivc.trace_settings.structure = TraceStructure::CLIENT_IVC_BENCH;

MockCircuitProducer circuit_producer;

Expand Down Expand Up @@ -89,15 +89,15 @@ TEST_F(ClientIVCIntegrationTests, ConsecutiveKernels)
TEST_F(ClientIVCIntegrationTests, BenchmarkCasePrecomputedVKs)
{
ClientIVC ivc;
ivc.trace_structure = TraceStructure::CLIENT_IVC_BENCH;
ivc.trace_settings.structure = TraceStructure::CLIENT_IVC_BENCH;

size_t NUM_CIRCUITS = 6;

// Precompute the verification keys for each circuit in the IVC
std::vector<std::shared_ptr<VerificationKey>> precomputed_vks;
{
MockCircuitProducer circuit_producer;
precomputed_vks = circuit_producer.precompute_verification_keys(NUM_CIRCUITS, ivc.trace_structure);
precomputed_vks = circuit_producer.precompute_verification_keys(NUM_CIRCUITS, ivc.trace_settings);
}

MockCircuitProducer circuit_producer;
Expand All @@ -123,7 +123,7 @@ TEST_F(ClientIVCIntegrationTests, BenchmarkCasePrecomputedVKs)
TEST_F(ClientIVCIntegrationTests, DatabusFailure)
{
ClientIVC ivc;
ivc.trace_structure = TraceStructure::CLIENT_IVC_BENCH;
ivc.trace_settings.structure = TraceStructure::CLIENT_IVC_BENCH;

MockCircuitProducer circuit_producer;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,10 +135,10 @@ class PrivateFunctionExecutionMockCircuitProducer {
* @param trace_structure Trace structuring must be known in advance because it effects the VKs
* @return set of num_circuits-many verification keys
*/
auto precompute_verification_keys(const size_t num_circuits, TraceStructure trace_structure)
auto precompute_verification_keys(const size_t num_circuits, TraceSettings trace_settings)
{
ClientIVC ivc; // temporary IVC instance needed to produce the complete kernel circuits
ivc.trace_structure = trace_structure;
ivc.trace_settings = trace_settings;

std::vector<std::shared_ptr<VerificationKey>> vkeys;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ class MockKernelTest : public ::testing::Test {
TEST_F(MockKernelTest, PinFoldingKernelSizes)
{
ClientIVC ivc;
ivc.trace_structure = TraceStructure::CLIENT_IVC_BENCH;
ivc.trace_settings.structure = TraceStructure::CLIENT_IVC_BENCH;

MockCircuitProducer circuit_producer;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -402,7 +402,7 @@ TEST_P(AcirIntegrationFoldingTest, DISABLED_FoldAndVerifyProgramStack)

ClientIVC ivc;
ivc.auto_verify_mode = true;
ivc.trace_structure = TraceStructure::SMALL_TEST;
ivc.trace_settings.structure = TraceStructure::SMALL_TEST;

while (!program_stack.empty()) {
auto program = program_stack.back();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ class IvcRecursionConstraintTest : public ::testing::Test {
TEST_F(IvcRecursionConstraintTest, AccumulateTwo)
{
ClientIVC ivc;
ivc.trace_structure = TraceStructure::SMALL_TEST;
ivc.trace_settings.structure = TraceStructure::SMALL_TEST;

// construct a mock app_circuit
Builder app_circuit = construct_mock_app_circuit(ivc);
Expand All @@ -185,7 +185,7 @@ TEST_F(IvcRecursionConstraintTest, AccumulateTwo)
TEST_F(IvcRecursionConstraintTest, AccumulateFour)
{
ClientIVC ivc;
ivc.trace_structure = TraceStructure::SMALL_TEST;
ivc.trace_settings.structure = TraceStructure::SMALL_TEST;

// construct a mock app_circuit
Builder app_circuit_0 = construct_mock_app_circuit(ivc);
Expand Down Expand Up @@ -232,7 +232,7 @@ TEST_F(IvcRecursionConstraintTest, AccumulateTwoFailure)
VerifierInputs alternative_verification_queue_entry;
{
ClientIVC ivc;
ivc.trace_structure = TraceStructure::SMALL_TEST;
ivc.trace_settings.structure = TraceStructure::SMALL_TEST;

// construct and accumulate a mock app circuit with a single unique public input
Builder app_circuit = construct_mock_app_circuit(ivc);
Expand All @@ -254,7 +254,7 @@ TEST_F(IvcRecursionConstraintTest, AccumulateTwoFailure)
// valid) witnesses during constraint system construction VS recursive verifier construction.

ClientIVC ivc;
ivc.trace_structure = TraceStructure::SMALL_TEST;
ivc.trace_settings.structure = TraceStructure::SMALL_TEST;

// construct and accumulate a mock app circuit with a single unique public input
Builder app_circuit = construct_mock_app_circuit(ivc);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ WASM_EXPORT void acir_fold_and_verify_program_stack(uint8_t const* acir_vec,

ClientIVC ivc;
ivc.auto_verify_mode = true;
ivc.trace_structure = TraceStructure::SMALL_TEST;
ivc.trace_settings.structure = TraceStructure::SMALL_TEST;

bool is_kernel = false;
while (!program_stack.empty()) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -703,7 +703,7 @@ TEST_F(join_split_tests, test_0_input_notes_and_detect_circuit_change)
// The below part detects any changes in the join-split circuit
constexpr size_t DYADIC_CIRCUIT_SIZE = 1 << 16;

constexpr uint256_t CIRCUIT_HASH("0x2b30566e4d921ea9b0c76802d86ea5b8381ffa78ef143af1b0d0e3045862cb6b");
constexpr uint256_t CIRCUIT_HASH("0xe833d0ffaa99f39ca31ef79ded0089b0ec614eb4a528c4e1c4af6221a003fa6e");
ledwards2225 marked this conversation as resolved.
Show resolved Hide resolved

const uint256_t circuit_hash = circuit.hash_circuit();
// circuit is finalized now
Expand Down
2 changes: 1 addition & 1 deletion barretenberg/cpp/src/barretenberg/flavor/flavor.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -360,7 +360,7 @@ template <typename T>
concept IsUltraPlonkFlavor = IsAnyOf<T, plonk::flavor::Ultra, UltraKeccakFlavor>;

template <typename T>
concept IsUltraPlonkOrHonk = IsAnyOf<T, plonk::flavor::Ultra, UltraFlavor, UltraKeccakFlavor, UltraFlavorWithZK, MegaFlavor>;
concept IsUltraPlonkOrHonk = IsAnyOf<T, plonk::flavor::Ultra, UltraFlavor, UltraKeccakFlavor, UltraFlavorWithZK, MegaFlavor, MegaZKFlavor>;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I presume this might have been a culprit for why a test was failing for the ZK flavor

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This concept should be called IsLiterallyAnything


template <typename T>
concept IsHonkFlavor = IsAnyOf<T, UltraFlavor, UltraKeccakFlavor, UltraFlavorWithZK, MegaFlavor, MegaZKFlavor>;
Expand Down
Loading
Loading