Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Typed encrypted and unencrypted L2 log containers #5422

Merged
merged 4 commits into from
Mar 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions yarn-project/archiver/src/archiver/archiver.test.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { Body, L2Block, L2BlockL2Logs, LogType } from '@aztec/circuit-types';
import { Body, EncryptedL2BlockL2Logs, L2Block, LogType, UnencryptedL2BlockL2Logs } from '@aztec/circuit-types';
import { EthAddress } from '@aztec/foundation/eth-address';
import { Fr } from '@aztec/foundation/fields';
import { sleep } from '@aztec/foundation/sleep';
Expand Down Expand Up @@ -102,7 +102,7 @@ describe('Archiver', () => {

for (const [index, x] of blockNumbers.entries()) {
const expectedTotalNumEncryptedLogs = 4 * x * (x * 2);
const totalNumEncryptedLogs = L2BlockL2Logs.unrollLogs([encryptedLogs[index]]).length;
const totalNumEncryptedLogs = EncryptedL2BlockL2Logs.unrollLogs([encryptedLogs[index]]).length;
expect(totalNumEncryptedLogs).toEqual(expectedTotalNumEncryptedLogs);
}

Expand All @@ -111,7 +111,7 @@ describe('Archiver', () => {

blockNumbers.forEach((x, index) => {
const expectedTotalNumUnencryptedLogs = 4 * (x + 1) * (x * 3);
const totalNumUnencryptedLogs = L2BlockL2Logs.unrollLogs([unencryptedLogs[index]]).length;
const totalNumUnencryptedLogs = UnencryptedL2BlockL2Logs.unrollLogs([unencryptedLogs[index]]).length;
expect(totalNumUnencryptedLogs).toEqual(expectedTotalNumUnencryptedLogs);
});

Expand Down
10 changes: 7 additions & 3 deletions yarn-project/archiver/src/archiver/archiver.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import {
FromLogType,
GetUnencryptedLogsResponse,
L1ToL2MessageSource,
L2Block,
Expand Down Expand Up @@ -276,8 +277,7 @@ export class Archiver implements ArchiveSource {
retrievedBlocks.retrievedData.map(async block => {
const blockLogs = block.body.txEffects
.flatMap(txEffect => (txEffect ? [txEffect.unencryptedLogs] : []))
.flatMap(txLog => txLog.unrollLogs())
.map(log => UnencryptedL2Log.fromBuffer(log));
.flatMap(txLog => txLog.unrollLogs());
await this.storeRegisteredContractClasses(blockLogs, block.number);
await this.storeDeployedContractInstances(blockLogs, block.number);
await this.storeBroadcastedIndividualFunctions(blockLogs, block.number);
Expand Down Expand Up @@ -437,7 +437,11 @@ export class Archiver implements ArchiveSource {
* @param logType - Specifies whether to return encrypted or unencrypted logs.
* @returns The requested logs.
*/
public getLogs(from: number, limit: number, logType: LogType): Promise<L2BlockL2Logs[]> {
public getLogs<TLogType extends LogType>(
from: number,
limit: number,
logType: TLogType,
): Promise<L2BlockL2Logs<FromLogType<TLogType>>[]> {
return this.store.getLogs(from, limit, logType);
}

Expand Down
13 changes: 10 additions & 3 deletions yarn-project/archiver/src/archiver/archiver_store.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
import {
Body,
EncryptedL2BlockL2Logs,
FromLogType,
GetUnencryptedLogsResponse,
InboxLeaf,
L2Block,
Expand All @@ -9,6 +11,7 @@ import {
TxEffect,
TxHash,
TxReceipt,
UnencryptedL2BlockL2Logs,
} from '@aztec/circuit-types';
import { Fr } from '@aztec/circuits.js';
import { AztecAddress } from '@aztec/foundation/aztec-address';
Expand Down Expand Up @@ -88,8 +91,8 @@ export interface ArchiverDataStore {
* @returns True if the operation is successful.
*/
addLogs(
encryptedLogs: L2BlockL2Logs | undefined,
unencryptedLogs: L2BlockL2Logs | undefined,
encryptedLogs: EncryptedL2BlockL2Logs | undefined,
unencryptedLogs: UnencryptedL2BlockL2Logs | undefined,
blockNumber: number,
): Promise<boolean>;

Expand Down Expand Up @@ -122,7 +125,11 @@ export interface ArchiverDataStore {
* @param logType - Specifies whether to return encrypted or unencrypted logs.
* @returns The requested logs.
*/
getLogs(from: number, limit: number, logType: LogType): Promise<L2BlockL2Logs[]>;
getLogs<TLogType extends LogType>(
from: number,
limit: number,
logType: TLogType,
): Promise<L2BlockL2Logs<FromLogType<TLogType>>[]>;

/**
* Gets unencrypted logs based on the provided filter.
Expand Down
12 changes: 5 additions & 7 deletions yarn-project/archiver/src/archiver/archiver_store_test_suite.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { InboxLeaf, L2Block, L2BlockContext, LogId, LogType, TxHash, UnencryptedL2Log } from '@aztec/circuit-types';
import { InboxLeaf, L2Block, L2BlockContext, LogId, LogType, TxHash } from '@aztec/circuit-types';
import '@aztec/circuit-types/jest';
import { AztecAddress, Fr, INITIAL_L2_BLOCK_NUM, L1_TO_L2_MSG_SUBTREE_HEIGHT } from '@aztec/circuits.js';
import {
Expand Down Expand Up @@ -367,11 +367,10 @@ export function describeArchiverDataStore(testName: string, getStore: () => Arch
const targetTxIndex = randomInt(txsPerBlock);
const targetFunctionLogIndex = randomInt(numPublicFunctionCalls);
const targetLogIndex = randomInt(numUnencryptedLogs);
const targetContractAddress = UnencryptedL2Log.fromBuffer(
const targetContractAddress =
blocks.retrievedData[targetBlockIndex].body.txEffects[targetTxIndex].unencryptedLogs.functionLogs[
targetFunctionLogIndex
].logs[targetLogIndex],
).contractAddress;
].logs[targetLogIndex].contractAddress;

const response = await store.getUnencryptedLogs({ contractAddress: targetContractAddress });

Expand All @@ -388,11 +387,10 @@ export function describeArchiverDataStore(testName: string, getStore: () => Arch
const targetTxIndex = randomInt(txsPerBlock);
const targetFunctionLogIndex = randomInt(numPublicFunctionCalls);
const targetLogIndex = randomInt(numUnencryptedLogs);
const targetSelector = UnencryptedL2Log.fromBuffer(
const targetSelector =
blocks.retrievedData[targetBlockIndex].body.txEffects[targetTxIndex].unencryptedLogs.functionLogs[
targetFunctionLogIndex
].logs[targetLogIndex],
).selector;
].logs[targetLogIndex].selector;

const response = await store.getUnencryptedLogs({ selector: targetSelector });

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
import {
Body,
EncryptedL2BlockL2Logs,
FromLogType,
GetUnencryptedLogsResponse,
InboxLeaf,
L2Block,
Expand All @@ -9,6 +11,7 @@ import {
TxEffect,
TxHash,
TxReceipt,
UnencryptedL2BlockL2Logs,
} from '@aztec/circuit-types';
import { Fr } from '@aztec/circuits.js';
import { AztecAddress } from '@aztec/foundation/aztec-address';
Expand Down Expand Up @@ -150,8 +153,8 @@ export class KVArchiverDataStore implements ArchiverDataStore {
* @returns True if the operation is successful.
*/
addLogs(
encryptedLogs: L2BlockL2Logs | undefined,
unencryptedLogs: L2BlockL2Logs | undefined,
encryptedLogs: EncryptedL2BlockL2Logs | undefined,
unencryptedLogs: UnencryptedL2BlockL2Logs | undefined,
blockNumber: number,
): Promise<boolean> {
return this.#logStore.addLogs(encryptedLogs, unencryptedLogs, blockNumber);
Expand Down Expand Up @@ -196,7 +199,11 @@ export class KVArchiverDataStore implements ArchiverDataStore {
* @param logType - Specifies whether to return encrypted or unencrypted logs.
* @returns The requested logs.
*/
getLogs(start: number, limit: number, logType: LogType): Promise<L2BlockL2Logs[]> {
getLogs<TLogType extends LogType>(
start: number,
limit: number,
logType: TLogType,
): Promise<L2BlockL2Logs<FromLogType<TLogType>>[]> {
try {
return Promise.resolve(Array.from(this.#logStore.getLogs(start, limit, logType)));
} catch (err) {
Expand Down
33 changes: 23 additions & 10 deletions yarn-project/archiver/src/archiver/kv_archiver_store/log_store.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
import {
EncryptedL2BlockL2Logs,
ExtendedUnencryptedL2Log,
FromLogType,
GetUnencryptedLogsResponse,
L2BlockL2Logs,
LogFilter,
LogId,
LogType,
UnencryptedL2BlockL2Logs,
UnencryptedL2Log,
} from '@aztec/circuit-types';
import { INITIAL_L2_BLOCK_NUM } from '@aztec/circuits.js/constants';
Expand Down Expand Up @@ -37,8 +40,8 @@ export class LogStore {
* @returns True if the operation is successful.
*/
addLogs(
encryptedLogs: L2BlockL2Logs | undefined,
unencryptedLogs: L2BlockL2Logs | undefined,
encryptedLogs: EncryptedL2BlockL2Logs | undefined,
unencryptedLogs: UnencryptedL2BlockL2Logs | undefined,
blockNumber: number,
): Promise<boolean> {
return this.db.transaction(() => {
Expand All @@ -61,10 +64,15 @@ export class LogStore {
* @param logType - Specifies whether to return encrypted or unencrypted logs.
* @returns The requested logs.
*/
*getLogs(start: number, limit: number, logType: LogType): IterableIterator<L2BlockL2Logs> {
*getLogs<TLogType extends LogType>(
start: number,
limit: number,
logType: TLogType,
): IterableIterator<L2BlockL2Logs<FromLogType<TLogType>>> {
const logMap = logType === LogType.ENCRYPTED ? this.#encryptedLogs : this.#unencryptedLogs;
const L2BlockL2Logs = logType === LogType.ENCRYPTED ? EncryptedL2BlockL2Logs : UnencryptedL2BlockL2Logs;
for (const buffer of logMap.values({ start, limit })) {
yield L2BlockL2Logs.fromBuffer(buffer);
yield L2BlockL2Logs.fromBuffer(buffer) as L2BlockL2Logs<FromLogType<TLogType>>;
}
}

Expand Down Expand Up @@ -94,7 +102,7 @@ export class LogStore {
}

const unencryptedLogsInBlock = this.#getBlockLogs(blockNumber, LogType.UNENCRYPTED);
const txLogs = unencryptedLogsInBlock.txLogs[txIndex].unrollLogs().map(log => UnencryptedL2Log.fromBuffer(log));
const txLogs = unencryptedLogsInBlock.txLogs[txIndex].unrollLogs();

const logs: ExtendedUnencryptedL2Log[] = [];
const maxLogsHit = this.#accumulateLogs(logs, blockNumber, txIndex, txLogs, filter);
Expand All @@ -118,9 +126,9 @@ export class LogStore {

let maxLogsHit = false;
loopOverBlocks: for (const [blockNumber, logBuffer] of this.#unencryptedLogs.entries({ start, end })) {
const unencryptedLogsInBlock = L2BlockL2Logs.fromBuffer(logBuffer);
const unencryptedLogsInBlock = UnencryptedL2BlockL2Logs.fromBuffer(logBuffer);
for (let txIndex = filter.afterLog?.txIndex ?? 0; txIndex < unencryptedLogsInBlock.txLogs.length; txIndex++) {
const txLogs = unencryptedLogsInBlock.txLogs[txIndex].unrollLogs().map(log => UnencryptedL2Log.fromBuffer(log));
const txLogs = unencryptedLogsInBlock.txLogs[txIndex].unrollLogs();
maxLogsHit = this.#accumulateLogs(logs, blockNumber, txIndex, txLogs, filter);
if (maxLogsHit) {
this.#log(`Max logs hit at block ${blockNumber}`);
Expand Down Expand Up @@ -161,14 +169,19 @@ export class LogStore {
return maxLogsHit;
}

#getBlockLogs(blockNumber: number, logType: LogType): L2BlockL2Logs {
#getBlockLogs<TLogType extends LogType>(
blockNumber: number,
logType: TLogType,
): L2BlockL2Logs<FromLogType<TLogType>> {
const logMap = logType === LogType.ENCRYPTED ? this.#encryptedLogs : this.#unencryptedLogs;
const L2BlockL2Logs: typeof EncryptedL2BlockL2Logs | typeof UnencryptedL2BlockL2Logs =
logType === LogType.ENCRYPTED ? EncryptedL2BlockL2Logs : UnencryptedL2BlockL2Logs;
const buffer = logMap.get(blockNumber);

if (!buffer) {
return new L2BlockL2Logs([]);
return new L2BlockL2Logs([]) as L2BlockL2Logs<FromLogType<TLogType>>;
}

return L2BlockL2Logs.fromBuffer(buffer);
return L2BlockL2Logs.fromBuffer(buffer) as L2BlockL2Logs<FromLogType<TLogType>>;
}
}
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import {
Body,
EncryptedL2BlockL2Logs,
ExtendedUnencryptedL2Log,
FromLogType,
GetUnencryptedLogsResponse,
InboxLeaf,
L2Block,
Expand All @@ -13,7 +15,7 @@ import {
TxHash,
TxReceipt,
TxStatus,
UnencryptedL2Log,
UnencryptedL2BlockL2Logs,
} from '@aztec/circuit-types';
import { Fr, INITIAL_L2_BLOCK_NUM } from '@aztec/circuits.js';
import { AztecAddress } from '@aztec/foundation/aztec-address';
Expand Down Expand Up @@ -51,13 +53,13 @@ export class MemoryArchiverStore implements ArchiverDataStore {
* An array containing all the encrypted logs that have been fetched so far.
* Note: Index in the "outer" array equals to (corresponding L2 block's number - INITIAL_L2_BLOCK_NUM).
*/
private encryptedLogsPerBlock: L2BlockL2Logs[] = [];
private encryptedLogsPerBlock: EncryptedL2BlockL2Logs[] = [];

/**
* An array containing all the unencrypted logs that have been fetched so far.
* Note: Index in the "outer" array equals to (corresponding L2 block's number - INITIAL_L2_BLOCK_NUM).
*/
private unencryptedLogsPerBlock: L2BlockL2Logs[] = [];
private unencryptedLogsPerBlock: UnencryptedL2BlockL2Logs[] = [];

/**
* Contains all L1 to L2 messages.
Expand Down Expand Up @@ -183,7 +185,11 @@ export class MemoryArchiverStore implements ArchiverDataStore {
* @param blockNumber - The block for which to add the logs.
* @returns True if the operation is successful.
*/
addLogs(encryptedLogs: L2BlockL2Logs, unencryptedLogs: L2BlockL2Logs, blockNumber: number): Promise<boolean> {
addLogs(
encryptedLogs: EncryptedL2BlockL2Logs,
unencryptedLogs: UnencryptedL2BlockL2Logs,
blockNumber: number,
): Promise<boolean> {
if (encryptedLogs) {
this.encryptedLogsPerBlock[blockNumber - INITIAL_L2_BLOCK_NUM] = encryptedLogs;
}
Expand Down Expand Up @@ -288,11 +294,17 @@ export class MemoryArchiverStore implements ArchiverDataStore {
* @param logType - Specifies whether to return encrypted or unencrypted logs.
* @returns The requested logs.
*/
getLogs(from: number, limit: number, logType: LogType): Promise<L2BlockL2Logs[]> {
getLogs<TLogType extends LogType>(
from: number,
limit: number,
logType: TLogType,
): Promise<L2BlockL2Logs<FromLogType<TLogType>>[]> {
if (from < INITIAL_L2_BLOCK_NUM || limit < 1) {
throw new Error(`Invalid limit: ${limit}`);
}
const logs = logType === LogType.ENCRYPTED ? this.encryptedLogsPerBlock : this.unencryptedLogsPerBlock;
const logs = (
logType === LogType.ENCRYPTED ? this.encryptedLogsPerBlock : this.unencryptedLogsPerBlock
) as L2BlockL2Logs<FromLogType<TLogType>>[];
if (from > logs.length) {
return Promise.resolve([]);
}
Expand Down Expand Up @@ -355,7 +367,7 @@ export class MemoryArchiverStore implements ArchiverDataStore {
const blockContext = this.l2BlockContexts[fromBlockIndex];
const blockLogs = this.unencryptedLogsPerBlock[fromBlockIndex];
for (; txIndexInBlock < blockLogs.txLogs.length; txIndexInBlock++) {
const txLogs = blockLogs.txLogs[txIndexInBlock].unrollLogs().map(log => UnencryptedL2Log.fromBuffer(log));
const txLogs = blockLogs.txLogs[txIndexInBlock].unrollLogs();
for (; logIndexInTx < txLogs.length; logIndexInTx++) {
const log = txLogs[logIndexInTx];
if (
Expand Down
8 changes: 5 additions & 3 deletions yarn-project/archiver/src/rpc/archiver_client.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
import {
EncryptedL2BlockL2Logs,
ExtendedUnencryptedL2Log,
L2Block,
L2BlockL2Logs,
NullifierMembershipWitness,
TxReceipt,
UnencryptedL2BlockL2Logs,
} from '@aztec/circuit-types';
import { EthAddress, Fr } from '@aztec/circuits.js';
import { createJsonRpcClient, makeFetch } from '@aztec/foundation/json-rpc/client';
Expand All @@ -18,10 +19,11 @@ export const createArchiverClient = (url: string, fetch = makeFetch([1, 2, 3], t
ExtendedUnencryptedL2Log,
Fr,
L2Block,
L2BlockL2Logs,
EncryptedL2BlockL2Logs,
UnencryptedL2BlockL2Logs,
},
{ TxReceipt, NullifierMembershipWitness },
false,
'archiver',
fetch,
);
) as ArchiveSource;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know why we suddenly need the cast here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The RemoteObject wrapper was failing to handle the conditional return type for getLogs

6 changes: 4 additions & 2 deletions yarn-project/archiver/src/rpc/archiver_server.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
import {
EncryptedL2BlockL2Logs,
ExtendedUnencryptedL2Log,
L2Block,
L2BlockL2Logs,
NullifierMembershipWitness,
TxEffect,
TxReceipt,
UnencryptedL2BlockL2Logs,
} from '@aztec/circuit-types';
import { EthAddress, Fr } from '@aztec/circuits.js';
import { JsonRpcServer } from '@aztec/foundation/json-rpc/server';
Expand All @@ -24,7 +25,8 @@ export function createArchiverRpcServer(archiverService: Archiver): JsonRpcServe
ExtendedUnencryptedL2Log,
Fr,
L2Block,
L2BlockL2Logs,
EncryptedL2BlockL2Logs,
UnencryptedL2BlockL2Logs,
TxEffect,
},
{ TxReceipt, NullifierMembershipWitness },
Expand Down
Loading
Loading