Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Dynamic proving #5346

Merged
merged 50 commits into from
Mar 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
2911491
Move block building to prover-client
PhilWindle Mar 15, 2024
95619a9
yarn prepare
PhilWindle Mar 15, 2024
fed1a40
Merge branch 'master' into pw/proving-2
PhilWindle Mar 15, 2024
45d1763
Fixes
PhilWindle Mar 15, 2024
a7b36ad
lockfile
PhilWindle Mar 15, 2024
9823ae6
Fixes
PhilWindle Mar 15, 2024
5a61d42
Fixes
PhilWindle Mar 17, 2024
c4591ea
Fix
PhilWindle Mar 17, 2024
508878f
Fix
PhilWindle Mar 17, 2024
1164281
Merge branch 'master' into pw/proving-2
PhilWindle Mar 17, 2024
a1569e0
Fixes
PhilWindle Mar 17, 2024
9710298
Merge fix
PhilWindle Mar 17, 2024
80419ca
Formatting
PhilWindle Mar 18, 2024
7a0546e
Formatting
PhilWindle Mar 18, 2024
c86ccdc
WIP
PhilWindle Mar 18, 2024
a2329eb
Merge branch 'pw/proving-2' of github.com:AztecProtocol/aztec-package…
PhilWindle Mar 18, 2024
132fe2c
WIP
PhilWindle Mar 20, 2024
6622591
WIP
PhilWindle Mar 20, 2024
d1e5bb7
WIP
PhilWindle Mar 20, 2024
faa6e77
Merge branch 'master' into pw/proving-2
PhilWindle Mar 20, 2024
dd5999b
Merge fixes
PhilWindle Mar 20, 2024
6815180
Merge fixes
PhilWindle Mar 20, 2024
dd6b059
WIP
PhilWindle Mar 20, 2024
8b5615d
Fixes
PhilWindle Mar 20, 2024
c9997cc
Comments
PhilWindle Mar 20, 2024
2253a7a
Fixes
PhilWindle Mar 21, 2024
55d8194
Fixes
PhilWindle Mar 21, 2024
769d298
Fix
PhilWindle Mar 21, 2024
7c56ae9
yarn prepare
PhilWindle Mar 21, 2024
0bbd504
Just using WASM for now
PhilWindle Mar 21, 2024
731079a
Fixes
PhilWindle Mar 21, 2024
03e7c13
Formatting
PhilWindle Mar 21, 2024
8c52b6e
Introduce proving status
PhilWindle Mar 21, 2024
ea6f053
Merge branch 'master' into pw/proving-2
PhilWindle Mar 21, 2024
21bc20f
Review changes
PhilWindle Mar 21, 2024
8b2d9ae
Merge branch 'master' into pw/proving-2
PhilWindle Mar 21, 2024
5fc7846
Merge fixes
PhilWindle Mar 21, 2024
0ba12ab
Review changes
PhilWindle Mar 21, 2024
2e3224c
Review changes
PhilWindle Mar 21, 2024
5c28f6c
Fixes
PhilWindle Mar 21, 2024
202b7f9
Logging
PhilWindle Mar 21, 2024
20f5df6
Merge branch 'master' into pw/proving-2
PhilWindle Mar 21, 2024
d2ef12a
Fixes
PhilWindle Mar 21, 2024
3e4897f
Merge branch 'master' into pw/proving-2
PhilWindle Mar 21, 2024
c4e1f9d
Merge fixes
PhilWindle Mar 21, 2024
c4e1fd4
Merge branch 'master' into pw/proving-2
PhilWindle Mar 21, 2024
27dc524
WIP
PhilWindle Mar 21, 2024
19b82d4
Merge branch 'master' into pw/proving-2
PhilWindle Mar 21, 2024
baf33ff
Fix
PhilWindle Mar 22, 2024
9890edc
Re-instate native simulation
PhilWindle Mar 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ cd ~/.aztec && docker-compose up
If you wish to run components of the Aztec network stack separately, you can use the `aztec start` command with various options for enabling components.

```bash
aztec start --node [nodeOptions] --pxe [pxeOptions] --archiver [archiverOptions] --sequencer [sequencerOptions] ----p2p-bootstrap [p2pOptions]
aztec start --node [nodeOptions] --pxe [pxeOptions] --archiver [archiverOptions] --sequencer [sequencerOptions] --prover [proverOptions] ----p2p-bootstrap [p2pOptions]
```

Starting the aztec node alongside a PXE, sequencer or archiver, will attach the components to the node. If you want to e.g. run a PXE separately to a node, you can:
Expand Down
2 changes: 2 additions & 0 deletions yarn-project/aztec-node/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,9 @@
"@aztec/l1-artifacts": "workspace:^",
"@aztec/merkle-tree": "workspace:^",
"@aztec/p2p": "workspace:^",
"@aztec/prover-client": "workspace:^",
"@aztec/sequencer-client": "workspace:^",
"@aztec/simulator": "workspace:^",
"@aztec/types": "workspace:^",
"@aztec/world-state": "workspace:^",
"koa": "^2.14.2",
Expand Down
6 changes: 5 additions & 1 deletion yarn-project/aztec-node/src/aztec-node/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ export type AztecNodeConfig = ArchiverConfig &
/** Whether the sequencer is disabled for this node. */
disableSequencer: boolean;

/** Whether the prover is disabled for this node. */
disableProver: boolean;

/** A URL for an archiver service that the node will use. */
archiverUrl?: string;
};
Expand All @@ -21,13 +24,14 @@ export type AztecNodeConfig = ArchiverConfig &
* @returns A valid aztec node config.
*/
export function getConfigEnvVars(): AztecNodeConfig {
const { SEQ_DISABLED } = process.env;
const { SEQ_DISABLED, PROVER_DISABLED } = process.env;
const allEnvVars: AztecNodeConfig = {
...getSequencerVars(),
...getArchiverVars(),
...getP2PConfigEnvVars(),
...getWorldStateVars(),
disableSequencer: !!SEQ_DISABLED,
disableProver: !!PROVER_DISABLED,
archiverUrl: process.env.ARCHIVER_URL,
};

Expand Down
27 changes: 24 additions & 3 deletions yarn-project/aztec-node/src/aztec-node/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import {
LogType,
MerkleTreeId,
NullifierMembershipWitness,
ProverClient,
PublicDataWitness,
SequencerConfig,
SiblingPath,
Expand All @@ -20,6 +21,7 @@ import {
TxHash,
TxReceipt,
TxStatus,
partitionReverts,
} from '@aztec/circuit-types';
import {
ARCHIVE_HEIGHT,
Expand All @@ -44,14 +46,14 @@ import { AztecLmdbStore } from '@aztec/kv-store/lmdb';
import { initStoreForRollup, openTmpStore } from '@aztec/kv-store/utils';
import { SHA256Trunc, StandardTree } from '@aztec/merkle-tree';
import { AztecKVTxPool, P2P, createP2PClient } from '@aztec/p2p';
import { DummyProver, TxProver } from '@aztec/prover-client';
import {
GlobalVariableBuilder,
PublicProcessorFactory,
SequencerClient,
WASMSimulator,
getGlobalVariableBuilder,
partitionReverts,
} from '@aztec/sequencer-client';
import { WASMSimulator } from '@aztec/simulator';
import { ContractClassPublic, ContractDataSource, ContractInstanceWithAddress } from '@aztec/types/contracts';
import {
MerkleTrees,
Expand All @@ -62,6 +64,7 @@ import {
} from '@aztec/world-state';

import { AztecNodeConfig } from './config.js';
import { getSimulationProvider } from './simulator-factory.js';

/**
* The aztec node.
Expand All @@ -81,6 +84,7 @@ export class AztecNodeService implements AztecNode {
protected readonly version: number,
protected readonly globalVariableBuilder: GlobalVariableBuilder,
protected readonly merkleTreesDb: AztecKVStore,
private readonly prover: ProverClient,
private log = createDebugLogger('aztec:node'),
) {
const message =
Expand Down Expand Up @@ -139,10 +143,25 @@ export class AztecNodeService implements AztecNode {
// start both and wait for them to sync from the block source
await Promise.all([p2pClient.start(), worldStateSynchronizer.start()]);

// start the prover if we have been told to
const simulationProvider = await getSimulationProvider(config, log);
const prover = config.disableProver
? await DummyProver.new()
: await TxProver.new(config, worldStateSynchronizer, simulationProvider);

// now create the sequencer
const sequencer = config.disableSequencer
? undefined
: await SequencerClient.new(config, p2pClient, worldStateSynchronizer, archiver, archiver, archiver);
: await SequencerClient.new(
config,
p2pClient,
worldStateSynchronizer,
archiver,
archiver,
archiver,
prover,
simulationProvider,
);

return new AztecNodeService(
config,
Expand All @@ -158,6 +177,7 @@ export class AztecNodeService implements AztecNode {
config.version,
getGlobalVariableBuilder(config),
store,
prover,
log,
);
}
Expand Down Expand Up @@ -299,6 +319,7 @@ export class AztecNodeService implements AztecNode {
await this.p2pClient.stop();
await this.worldStateSynchronizer.stop();
await this.blockSource.stop();
await this.prover.stop();
this.log.info(`Stopped`);
}

Expand Down
24 changes: 24 additions & 0 deletions yarn-project/aztec-node/src/aztec-node/simulator-factory.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
import { DebugLogger } from '@aztec/foundation/log';
import { NativeACVMSimulator, SimulationProvider, WASMSimulator } from '@aztec/simulator';

import * as fs from 'fs/promises';

import { AztecNodeConfig } from './config.js';

export async function getSimulationProvider(
config: AztecNodeConfig,
logger?: DebugLogger,
): Promise<SimulationProvider> {
if (config.acvmBinaryPath && config.acvmWorkingDirectory) {
try {
await fs.access(config.acvmBinaryPath, fs.constants.R_OK);
await fs.mkdir(config.acvmWorkingDirectory, { recursive: true });
logger?.(`Using native ACVM at ${config.acvmBinaryPath} and working directory ${config.acvmWorkingDirectory}`);
return new NativeACVMSimulator(config.acvmWorkingDirectory, config.acvmBinaryPath);
} catch {
logger?.(`Failed to access ACVM at ${config.acvmBinaryPath}, falling back to WASM`);
}
}
logger?.('Using WASM ACVM simulation');
return new WASMSimulator();
}
2 changes: 1 addition & 1 deletion yarn-project/aztec-node/terraform/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ resource "aws_ecs_task_definition" "aztec-node" {
{
"name": "${var.DEPLOY_TAG}-aztec-node-${count.index + 1}",
"image": "${var.DOCKERHUB_ACCOUNT}/aztec:${var.DEPLOY_TAG}",
"command": ["start", "--node", "--archiver", "--sequencer"],
"command": ["start", "--node", "--archiver", "--sequencer", "--prover"],
"essential": true,
"memoryReservation": 3776,
"portMappings": [
Expand Down
6 changes: 6 additions & 0 deletions yarn-project/aztec-node/tsconfig.json
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,15 @@
{
"path": "../p2p"
},
{
"path": "../prover-client"
},
{
"path": "../sequencer-client"
},
{
"path": "../simulator"
},
{
"path": "../types"
},
Expand Down
1 change: 1 addition & 0 deletions yarn-project/aztec/src/cli/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ export function getProgram(userLog: LogFn, debugLogger: DebugLogger): Command {
.option('-px, --pxe [options]', cliTexts.pxe)
.option('-a, --archiver [options]', cliTexts.archiver)
.option('-s, --sequencer [options]', cliTexts.sequencer)
.option('-r, --prover [options]', cliTexts.prover)
.option('-p2p, --p2p-bootstrap [options]', cliTexts.p2pBootstrap)
.action(async options => {
// list of 'stop' functions to call when process ends
Expand Down
4 changes: 4 additions & 0 deletions yarn-project/aztec/src/cli/cmds/start_node.ts
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,10 @@ export const startNode = async (
nodeConfig.publisherPrivateKey = `0x${Buffer.from(privKey!).toString('hex')}`;
}

if (!options.prover) {
nodeConfig.disableProver = true;
}

// Create and start Aztec Node.
const node = await createAztecNode(nodeConfig);
const nodeServer = createAztecNodeRpcServer(node);
Expand Down
7 changes: 7 additions & 0 deletions yarn-project/aztec/src/cli/texts.ts
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,14 @@ export const cliTexts = {
'requiredConfirmations:SEQ_REQUIRED_CONFIRMATIONS - number - The number of confirmations required before publishing a block. Default: 1\n' +
'l1BlockPublishRetryIntervalMS:SEQ_PUBLISH_RETRY_INTERVAL_MS - number - The interval in ms to wait before retrying to publish a block. Default: 1000\n' +
'transactionPollingIntervalMS:SEQ_TX_POLLING_INTERVAL_MS - number - The interval in ms to wait before polling for new transactions. Default: 1000\n' +
'acvmBinaryPath:ACVM_BINARY_PATH - string - The full path to an instance of the acvm cli application. If not provided will fallback to WASM circuit simulation\n' +
'acvmWorkingDirectory:ACVM_WORKING_DIRECTORY - string - A directory to use for temporary files used by the acvm application. If not provided WASM circuit simulation will be used\n' +
contractAddresses,
prover:
'Starts a Prover with options. If started additionally to --node, the Prover will attach to that node.\n' +
'Available options are listed below as cliProperty:ENV_VARIABLE_NAME.\n' +
'acvmBinaryPath:ACVM_BINARY_PATH - string - The full path to an instance of the acvm cli application. If not provided will fallback to WASM circuit simulation\n' +
'acvmWorkingDirectory:ACVM_WORKING_DIRECTORY - string - A directory to use for temporary files used by the acvm application. If not provided WASM circuit simulation will be used\n',
p2pBootstrap:
'Starts a P2P bootstrap node with options.\n' +
'Available options are listed below as cliProperty:ENV_VARIABLE_NAME.\n' +
Expand Down
4 changes: 4 additions & 0 deletions yarn-project/circuit-types/src/body.ts
Original file line number Diff line number Diff line change
Expand Up @@ -95,4 +95,8 @@ export class Body {

return new Body(txEffects);
}

static empty() {
return new Body([]);
}
}
41 changes: 41 additions & 0 deletions yarn-project/circuit-types/src/interfaces/block-prover.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
import { Fr, GlobalVariables, Proof } from '@aztec/circuits.js';

import { L2Block } from '../l2_block.js';
import { ProcessedTx } from '../tx/processed_tx.js';

export enum PROVING_STATUS {
SUCCESS,
FAILURE,
}

export type ProvingSuccess = {
status: PROVING_STATUS.SUCCESS;
block: L2Block;
proof: Proof;
};

export type ProvingFailure = {
status: PROVING_STATUS.FAILURE;
reason: string;
};

export type ProvingResult = ProvingSuccess | ProvingFailure;

export type ProvingTicket = {
provingPromise: Promise<ProvingResult>;
};

/**
* The interface to the block prover.
* Provides the ability to generate proofs and build rollups.
*/
export interface BlockProver {
startNewBlock(
numTxs: number,
globalVariables: GlobalVariables,
l1ToL2Messages: Fr[],
emptyTx: ProcessedTx,
): Promise<ProvingTicket>;

addNewTx(tx: ProcessedTx): Promise<void>;
}
2 changes: 2 additions & 0 deletions yarn-project/circuit-types/src/interfaces/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,5 @@ export * from './sync-status.js';
export * from './configs.js';
export * from './nullifier_tree.js';
export * from './public_data_tree.js';
export * from './prover-client.js';
export * from './block-prover.js';
11 changes: 11 additions & 0 deletions yarn-project/circuit-types/src/interfaces/prover-client.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
import { BlockProver } from './block-prover.js';

/**
* The interface to the prover client.
* Provides the ability to generate proofs and build rollups.
*/
export interface ProverClient extends BlockProver {
start(): Promise<void>;

stop(): Promise<void>;
}
12 changes: 12 additions & 0 deletions yarn-project/circuit-types/src/l2_block.ts
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,18 @@ export class L2Block {
});
}

/**
* Creates an L2 block containing empty data.
* @returns The L2 block.
*/
static empty(): L2Block {
return L2Block.fromFields({
archive: AppendOnlyTreeSnapshot.zero(),
header: Header.empty(),
body: Body.empty(),
});
}

get number(): number {
return Number(this.header.globalVariables.blockNumber.toBigInt());
}
Expand Down
1 change: 1 addition & 0 deletions yarn-project/circuit-types/src/tx/index.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
export * from './tx.js';
export * from './tx_hash.js';
export * from './tx_receipt.js';
export * from './processed_tx.js';
12 changes: 12 additions & 0 deletions yarn-project/circuit-types/src/tx_effect.ts
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,18 @@ export class TxEffect {
);
}

static empty(): TxEffect {
return new TxEffect(
RevertCode.OK,
makeTuple(MAX_NEW_NOTE_HASHES_PER_TX, Fr.zero),
makeTuple(MAX_NEW_NULLIFIERS_PER_TX, Fr.zero),
makeTuple(MAX_NEW_L2_TO_L1_MSGS_PER_TX, Fr.zero),
makeTuple(MAX_PUBLIC_DATA_UPDATE_REQUESTS_PER_TX, PublicDataWrite.empty),
TxL2Logs.empty(),
TxL2Logs.empty(),
);
}

/**
* Returns a string representation of the TxEffect object.
*/
Expand Down
3 changes: 2 additions & 1 deletion yarn-project/circuits.js/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@
"./types": "./dest/types/index.js",
"./constants": "./dest/constants.gen.js",
"./contract": "./dest/contract/index.js",
"./merkle": "./dest/merkle/index.js"
"./merkle": "./dest/merkle/index.js",
"./simulation": "./dest/simulator/index.js"
},
"typedocOptions": {
"entryPoints": [
Expand Down
4 changes: 3 additions & 1 deletion yarn-project/circuits.js/src/structs/proof.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
import { BufferReader, serializeToBuffer } from '@aztec/foundation/serialize';

const EMPTY_PROOF_SIZE = 42;

/**
* The Proof class is a wrapper around the circuits proof.
* Underlying it is a buffer of proof data in a form a barretenberg prover understands.
Expand Down Expand Up @@ -47,5 +49,5 @@ export class Proof {
* @returns The empty "proof".
*/
export function makeEmptyProof() {
return new Proof(Buffer.alloc(0));
return new Proof(Buffer.alloc(EMPTY_PROOF_SIZE, 0));
}
3 changes: 3 additions & 0 deletions yarn-project/end-to-end/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,10 @@
"@aztec/noir-contracts.js": "workspace:^",
"@aztec/p2p": "workspace:^",
"@aztec/protocol-contracts": "workspace:^",
"@aztec/prover-client": "workspace:^",
"@aztec/pxe": "workspace:^",
"@aztec/sequencer-client": "workspace:^",
"@aztec/simulator": "workspace:^",
"@aztec/types": "workspace:^",
"@aztec/world-state": "workspace:^",
"@jest/globals": "^29.5.0",
Expand All @@ -50,6 +52,7 @@
"crypto-browserify": "^3.12.0",
"glob": "^10.3.10",
"jest": "^29.5.0",
"jest-mock-extended": "^3.0.5",
"koa": "^2.14.2",
"koa-static": "^5.0.0",
"levelup": "^5.1.1",
Expand Down
Loading
Loading