feat: gateway preparation #3006

perekopskiy · 2024-10-03T08:30:52Z

What ❔

adds new fields to DB tables and rust structs
adds new config variables
update commitment generator to work with post-gateway
adds new vm subversion (vm fast is not changed yet)

Why ❔

prepare for gateway, reduce sync-layer-stable diff

Checklist

PR title corresponds to the body of PR (we generate changelog entries from PRs).
Tests for the changes have been added / updated.
Documentation comments have been added / updated.
Code has been formatted via zk_supervisor fmt and zk_supervisor lint.

core/lib/multivm/src/versions/vm_latest/types/internals/pubdata.rs

core/lib/multivm/src/versions/vm_latest/bootloader_state/utils.rs

core/lib/multivm/src/versions/vm_latest/types/internals/pubdata.rs

core/node/commitment_generator/src/utils.rs

core/node/node_framework/src/implementations/layers/state_keeper/output_handler.rs

core/node/proof_data_handler/src/request_processor.rs

core/lib/types/src/block.rs

core/lib/multivm/src/versions/vm_latest/types/internals/pubdata.rs

core/lib/types/src/block.rs

core/lib/basic_types/src/protocol_version.rs

slowli

Not a full review, just some quick feedback. Will try to look at the PR more exhaustively tomorrow.

Not going to lie, the current PubdataBuilder design looks overly complicated, and in some aspects worse than the ad-hoc design during the last review. I think it could be salvaged though; see line comments.

core/lib/dal/src/consensus/mod.rs

core/lib/vm_interface/src/executor.rs

core/lib/vm_interface/src/pubdata/mod.rs

core/lib/vm_interface/src/vm.rs

core/lib/multivm/src/versions/vm_fast/vm.rs

core/lib/vm_executor/src/storage.rs

core/lib/vm_interface/Cargo.toml

core/lib/vm_interface/src/pubdata/mod.rs

perekopskiy · 2024-10-14T15:51:38Z

Not going to lie, the current PubdataBuilder design looks overly complicated, and in some aspects worse than the ad-hoc design during the last review. I think it could be salvaged though; see line comments.

I agree but the previous version was rather naive. I'm trying to find some spot between "keep it as simple as possible and delay dealing with complexity till we need first custom builder" and "implement complex solution that fully allows custom pubdata"

slowli

Sorry for merging #3109 which requires to fix edit conflicts in multivm unit tests (seems quite straightforward – you should move the changes to the test logic in versions::testonly submodules.

core/lib/multivm/src/dump.rs

core/lib/vm_interface/src/vm.rs

core/node/state_keeper/src/io/mempool.rs

core/node/state_keeper/src/io/persistence.rs

core/lib/vm_interface/src/utils/dump.rs

core/lib/vm_interface/src/vm.rs

core/node/commitment_generator/src/lib.rs

core/node/consensus/src/storage/store.rs

slowli

Can't say I've looked thoroughly at 100% of the code, but it looks good in general.

core/lib/vm_interface/src/types/inputs/execution_mode.rs

core/lib/vm_interface/src/utils/shadow.rs

core/node/commitment_generator/src/lib.rs

core/node/commitment_generator/src/utils.rs

core/lib/vm_interface/src/pubdata/mod.rs

github-actions · 2024-10-24T08:29:49Z

No performance difference detected (anymore)

🤖 I have created a release *beep* *boop* --- ## [16.6.0](prover-v16.5.0...prover-v16.6.0) (2024-10-31) ### Features * (DB migration) Rename recursion_scheduler_level_vk_hash to snark_wrapper_vk_hash ([#2809](#2809)) ([64f9551](64f9551)) * Add initial version prover_autoscaler ([#2993](#2993)) ([ebf9604](ebf9604)) * added seed_peers to consensus global config ([#2920](#2920)) ([e9d1d90](e9d1d90)) * attester committees data extractor (BFT-434) ([#2684](#2684)) ([92dde03](92dde03)) * Bump crypto and protocol deps ([#2825](#2825)) ([a5ffaf1](a5ffaf1)) * **circuit_prover:** Add circuit prover ([#2908](#2908)) ([48317e6](48317e6)) * **consensus:** Support for syncing blocks before consensus genesis over p2p network ([#3040](#3040)) ([d3edc3d](d3edc3d)) * **da-clients:** add secrets ([#2954](#2954)) ([f4631e4](f4631e4)) * gateway preparation ([#3006](#3006)) ([16f2757](16f2757)) * Integrate tracers and implement circuits tracer in vm2 ([#2653](#2653)) ([87b02e3](87b02e3)) * Move prover data to /home/popzxc/workspace/current/zksync-era/prover/data ([#2778](#2778)) ([62e4d46](62e4d46)) * Prover e2e test ([#2975](#2975)) ([0edd796](0edd796)) * **prover:** add CLI option to run prover with max allocation ([#2794](#2794)) ([35e4cae](35e4cae)) * **prover:** Add endpoint to PJM to get queue reports ([#2918](#2918)) ([2cec83f](2cec83f)) * **prover:** Add error to panic message of prover ([#2807](#2807)) ([6e057eb](6e057eb)) * **prover:** Add min_provers and dry_run features. Improve metrics and test. ([#3129](#3129)) ([7c28964](7c28964)) * **prover:** Add scale failure events watching and pods eviction. ([#3175](#3175)) ([dd166f8](dd166f8)) * **prover:** Add sending scale requests for Scaler targets ([#3194](#3194)) ([767c5bc](767c5bc)) * **prover:** Add support for scaling WGs and compressor ([#3179](#3179)) ([c41db9e](c41db9e)) * **prover:** Autoscaler sends scale request to appropriate agents. ([#3150](#3150)) ([bfedac0](bfedac0)) * **prover:** Extract keystore into a separate crate ([#2797](#2797)) ([e239260](e239260)) * **prover:** Optimize setup keys loading ([#2847](#2847)) ([19887ef](19887ef)) * **prover:** Refactor WitnessGenerator ([#2845](#2845)) ([934634b](934634b)) * **prover:** Update witness generator to zkevm_test_harness 0.150.6 ([#3029](#3029)) ([2151c28](2151c28)) * **prover:** Use query macro instead string literals for queries ([#2930](#2930)) ([1cf959d](1cf959d)) * **prover:** WG refactoring [#3](#3) ([#2942](#2942)) ([df68762](df68762)) * **prover:** WitnessGenerator refactoring [#2](#2) ([#2899](#2899)) ([36e5340](36e5340)) * Refactor metrics/make API use binaries ([#2735](#2735)) ([8ed086a](8ed086a)) * Remove prover db from house keeper ([#2795](#2795)) ([85b7346](85b7346)) * **tee:** use hex serialization for RPC responses ([#2887](#2887)) ([abe0440](abe0440)) * **utils:** Rework locate_workspace, introduce Workspace type ([#2830](#2830)) ([d256092](d256092)) * vm2 tracers can access storage ([#3114](#3114)) ([e466b52](e466b52)) * **vm:** Do not panic on VM divergence ([#2705](#2705)) ([7aa5721](7aa5721)) * **vm:** EVM emulator support – base ([#2979](#2979)) ([deafa46](deafa46)) * **vm:** Extract batch executor to separate crate ([#2702](#2702)) ([b82dfa4](b82dfa4)) * **zk_toolbox:** `zk_supervisor prover` subcommand ([#2820](#2820)) ([3506731](3506731)) * **zk_toolbox:** Add external_node consensus support ([#2821](#2821)) ([4a10d7d](4a10d7d)) * **zk_toolbox:** Add SQL format for zk supervisor ([#2950](#2950)) ([540e5d7](540e5d7)) * **zk_toolbox:** deploy legacy bridge ([#2837](#2837)) ([93b4e08](93b4e08)) * **zk_toolbox:** Redesign zk_toolbox commands ([#3003](#3003)) ([114834f](114834f)) * **zkstack_cli:** Build dependencies at zkstack build time ([#3157](#3157)) ([724d9a9](724d9a9)) ### Bug Fixes * allow compilation under current toolchain ([#3176](#3176)) ([89eadd3](89eadd3)) * **api:** Return correct flat call tracer ([#2917](#2917)) ([218646a](218646a)) * count SECP256 precompile to account validation gas limit as well ([#2859](#2859)) ([fee0c2a](fee0c2a)) * Fix Doc lint. ([#3158](#3158)) ([c79949b](c79949b)) * ignore unknown fields in rpc json response ([#2962](#2962)) ([692ea73](692ea73)) * **prover:** Do not exit on missing watcher data. ([#3119](#3119)) ([76ed6d9](76ed6d9)) * **prover:** fix setup_metadata_to_setup_data_key ([#2875](#2875)) ([4ae5a93](4ae5a93)) * **prover:** Run for zero queue to allow scaling down to 0 ([#3115](#3115)) ([bbe1919](bbe1919)) * **tee_verifier:** correctly initialize storage for re-execution ([#3017](#3017)) ([9d88373](9d88373)) * **vm:** Prepare new VM for use in API server and fix divergences ([#2994](#2994)) ([741b77e](741b77e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [25.1.0](core-v25.0.0...core-v25.1.0) (2024-11-04) ### Features * add `block.timestamp` asserter for AA ([#3031](#3031)) ([069d38d](069d38d)) * allow vm2 tracers to stop execution ([#3183](#3183)) ([9dae839](9dae839)) * **api:** get rid of tx receipt root ([#3187](#3187)) ([6c034f6](6c034f6)) * **api:** Integrate new VM into API server (no tracers) ([#3033](#3033)) ([8e75d4b](8e75d4b)) * base token integration tests ([#2509](#2509)) ([8db7e93](8db7e93)) * **consensus:** enabled syncing pregenesis blocks over p2p ([#3192](#3192)) ([6adb224](6adb224)) * **da-clients:** add Celestia client ([#2983](#2983)) ([d88b875](d88b875)) * **da-clients:** add EigenDA client ([#3155](#3155)) ([5161eed](5161eed)) * gateway preparation ([#3006](#3006)) ([16f2757](16f2757)) * Implement gas relay mode and inclusion data for data attestation ([#3070](#3070)) ([561fc1b](561fc1b)) * **metadata-calculator:** Add debug endpoints for tree API ([#3167](#3167)) ([3815252](3815252)) * **proof-data-handler:** add first processed batch option ([#3112](#3112)) ([1eb69d4](1eb69d4)) * **proof-data-handler:** add tee_proof_generation_timeout_in_secs param ([#3128](#3128)) ([f3724a7](f3724a7)) * **prover:** Add queue metric to report autoscaler view of the queue. ([#3206](#3206)) ([2721396](2721396)) * **prover:** Add sending scale requests for Scaler targets ([#3194](#3194)) ([767c5bc](767c5bc)) * **prover:** Add support for scaling WGs and compressor ([#3179](#3179)) ([c41db9e](c41db9e)) * **vm:** Support EVM emulation in fast VM ([#3163](#3163)) ([9ad1f0d](9ad1f0d)) ### Bug Fixes * **consensus:** better logging of errors ([#3170](#3170)) ([a5028da](a5028da)) * **consensus:** made attestation controller non-critical ([#3180](#3180)) ([6ee9f1f](6ee9f1f)) * **consensus:** payload encoding protected by protocol_version ([#3168](#3168)) ([8089b78](8089b78)) * **da-clients:** add padding to the data within EigenDA blob ([#3203](#3203)) ([8ae06b2](8ae06b2)) * **da-clients:** enable tls-roots feature for tonic ([#3201](#3201)) ([42f177a](42f177a)) * extend allowed storage slots for validation as per EIP-7562 ([#3166](#3166)) ([c76da16](c76da16)) * **merkle-tree:** Fix tree truncation ([#3178](#3178)) ([9654097](9654097)) * **tee_prover:** add prometheus pull listener ([#3169](#3169)) ([1ffd22f](1ffd22f)) * update logging in cbt l1 behaviour ([#3149](#3149)) ([d0f61b0](d0f61b0)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>

perekopskiy added 2 commits October 3, 2024 11:23

reduce sync-layer-stable diff

8546f1c

fix test

48fd0e6

perekopskiy requested review from RomanBrodetski, slowli and StanislavBreadless October 3, 2024 09:59

perekopskiy marked this pull request as ready for review October 3, 2024 09:59

slowli reviewed Oct 4, 2024

View reviewed changes

StanislavBreadless previously approved these changes Oct 4, 2024

View reviewed changes

perekopskiy added 3 commits October 7, 2024 14:58

nits

aff2e98

move bridge address validation to SK persistance

cd015b6

fix en

068d654

perekopskiy dismissed StanislavBreadless’s stale review via 068d654 October 7, 2024 12:15

fix

847c3da

StanislavBreadless previously approved these changes Oct 8, 2024

View reviewed changes

rework pubdata approach

f9e4c90

perekopskiy dismissed StanislavBreadless’s stale review via f9e4c90 October 8, 2024 13:41

perekopskiy added 2 commits October 8, 2024 16:44

unify naming

379b5c3

merge main

96f3af2

perekopskiy requested a review from slowli October 8, 2024 13:53

perekopskiy added 8 commits October 8, 2024 17:30

fix commitment

b00be9d

fix

ecd8e24

resolve conflicts

380c11d

update evm commitment test json

b0d6af7

another test json fix

8106288

Resolve conflicts

f7133a6

Merge branch 'main' into sync-layer-reduce-diff

d4f5d67

fix after merge

d8c6f4c

slowli reviewed Oct 14, 2024

View reviewed changes

Resolve conflicts

b8cee5e

perekopskiy added 3 commits October 18, 2024 12:40

update tests

0984283

fix tests

26bb438

update protocol version in json commitment test

a56e0ef

perekopskiy requested a review from slowli October 18, 2024 11:02

remove pubdata_params from oneshot env

4fbc3bb

slowli reviewed Oct 21, 2024

View reviewed changes

perekopskiy added 3 commits October 22, 2024 14:11

Resolve conflicts

8ade31b

remove finish_batch from dump playback

0825647

Address comments

b22e77c

perekopskiy requested a review from slowli October 22, 2024 13:27

slowli previously approved these changes Oct 22, 2024

View reviewed changes

nits

f8c6863

perekopskiy dismissed slowli’s stale review via f8c6863 October 22, 2024 17:48

slowli previously approved these changes Oct 23, 2024

View reviewed changes

perekopskiy added this pull request to the merge queue Oct 23, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 23, 2024

resolve conflicts

37d1e0f

perekopskiy dismissed slowli’s stale review via 37d1e0f October 23, 2024 17:55

fix bootloader tests

fadb376

slowli approved these changes Oct 24, 2024

View reviewed changes

perekopskiy added this pull request to the merge queue Oct 24, 2024

Merged via the queue into main with commit 16f2757 Oct 24, 2024
43 checks passed

perekopskiy deleted the sync-layer-reduce-diff branch October 24, 2024 09:34

This was referenced Oct 24, 2024

chore(main): release core 25.1.0 #3162

Merged

chore(main): release zkstack_cli 0.2.0 #3086

Open

chore(main): release prover 16.6.0 #2761

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: gateway preparation #3006

feat: gateway preparation #3006

perekopskiy commented Oct 3, 2024

slowli left a comment

perekopskiy commented Oct 14, 2024

slowli left a comment

slowli left a comment

github-actions bot commented Oct 24, 2024

feat: gateway preparation #3006

feat: gateway preparation #3006

Conversation

perekopskiy commented Oct 3, 2024

What ❔

Why ❔

Checklist

slowli left a comment

Choose a reason for hiding this comment

perekopskiy commented Oct 14, 2024

slowli left a comment

Choose a reason for hiding this comment

slowli left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 24, 2024

No performance difference detected (anymore)