feat(vm): Extract oneshot VM executor – environment types #2885
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
slowli
commented
Sep 16, 2024
slowli
commented
Sep 16, 2024
perekopskiy
previously approved these changes
Sep 19, 2024
There was a problem hiding this comment.
No additional concerns apart from #2885 (comment). Please fix here or create a follow up
perekopskiy
approved these changes
Sep 23, 2024
slowli
deleted the
aov-pla-1018-extract-oneshot-vm-executor-interface-env
branch
github-merge-queue bot
pushed a commit
that referenced
this pull request
Sep 23, 2024
🤖 I have created a release *beep* *boop* --- ## [24.26.0](core-v24.25.0...core-v24.26.0) (2024-09-23) ### Features * added seed_peers to consensus global config ([#2920](#2920)) ([e9d1d90](e9d1d90)) * **circuit_prover:** Add circuit prover ([#2908](#2908)) ([48317e6](48317e6)) * **prover:** Add endpoint to PJM to get queue reports ([#2918](#2918)) ([2cec83f](2cec83f)) * **vm:** Do not panic on VM divergence ([#2705](#2705)) ([7aa5721](7aa5721)) * **vm:** Extract oneshot VM executor – environment types ([#2885](#2885)) ([a2d4126](a2d4126)) ### Bug Fixes * **api_server:** fix blob_gas length ([#2673](#2673)) ([44a8f79](44a8f79)) * **eth-sender:** print better error message in case of missing blob prices ([#2927](#2927)) ([38fc824](38fc824)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>
Deniallugo
pushed a commit
that referenced
this pull request
Sep 24, 2024
## What ❔ Continues oneshot VM executor extraction by moving generic environment types to the `zksync_vm_executor` crate. ## Why ❔ Makes it possible to use the VM executor outside the API server. E.g., it's now used in `zksync_node_consensus`. ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Tests for the changes have been added / updated. - [x] Documentation comments have been added / updated. - [x] Code has been formatted via `zk fmt` and `zk lint`.
Deniallugo
pushed a commit
that referenced
this pull request
Sep 24, 2024
🤖 I have created a release *beep* *boop* --- ## [24.26.0](core-v24.25.0...core-v24.26.0) (2024-09-23) ### Features * added seed_peers to consensus global config ([#2920](#2920)) ([e9d1d90](e9d1d90)) * **circuit_prover:** Add circuit prover ([#2908](#2908)) ([48317e6](48317e6)) * **prover:** Add endpoint to PJM to get queue reports ([#2918](#2918)) ([2cec83f](2cec83f)) * **vm:** Do not panic on VM divergence ([#2705](#2705)) ([7aa5721](7aa5721)) * **vm:** Extract oneshot VM executor – environment types ([#2885](#2885)) ([a2d4126](a2d4126)) ### Bug Fixes * **api_server:** fix blob_gas length ([#2673](#2673)) ([44a8f79](44a8f79)) * **eth-sender:** print better error message in case of missing blob prices ([#2927](#2927)) ([38fc824](38fc824)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>
juan518munoz
added a commit
to lambdaclass/zksync-era
that referenced
this pull request
Oct 8, 2024
* feat(ci): Deprecate use of zk tool in docker-build process (#2901)
## What ❔
Remove use of ZK tool in docker-build process
## Why ❔
Improve readability of CI, speeds up building process
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* test: Investigate L1 "nonce too low" errors (#2848)
## What ❔
- Resolves "nonce too low" errors on L1 by retrying transaction
requests. This is a temporary measure until a new version of `reth` is
released.
- Reduces spammy logs for integration tests. Enables verbose output for
integration tests in CI.
- Fixes some other issues with tests (e.g., a data race in the snapshot
recovery test).
## Why ❔
Makes CI more stable.
## Checklist
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
* fix(ci): vm perf to prometheus (#2909)
## What ❔
<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
Signed-off-by: Danil <deniallugo@gmail.com>
* feat(ci): Add building base docker image (#2913)
## What ❔
Workflow to build base-image
## Why ❔
Added ability to push to ghcr.io
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* fix(ci): Fix multiarch for build-base CI (#2914)
## What ❔
Fix multiarch command for build-base-image CI
## Why ❔
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* fix(ci): Switch build-base to ghcr.io (#2912)
## What ❔
Change build-base image location from dockerhub to ghcr.io
## Why ❔
Workaround for dockerhub rate-limiting
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* feat: make `to` address optional for transaction data (#2852)
## What ❔
This PR is the first part of [this
one](https://github.com/matter-labs/zksync-era/pull/2631).
We decided to separate it into smaller PRs to make it easier to read and
review.
This PR contains only the changes to the `TransactionData` structs for
the different VM versions, modifying them for an optional field.
The other parts are currently available in our repo fork, as they are
based on this branch. The other PRs are:
- [feat: evm simulator as base system
contract](https://github.com/lambdaclass/zksync-era/pull/256):
integrates the EVM interpreter contract as a base system contract and
includes all necessary updates for zksync-era to run with it.
- [feat: make evm simulator optional for the
server](https://github.com/lambdaclass/zksync-era/pull/257): makes the
evm simulator optional to run using a config file.
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* feat(prover): WitnessGenerator refactoring #2 (#2899)
## What ❔
Introduce WitnessGenerator trait
Rename some methods
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* feat(tee): use hex serialization for RPC responses (#2887)
## What ❔
Following Anton's suggestion, we have switched to hex serialization for
API/RPC requests and responses. Previously, we used default JSON
serialization for Vec<u8>, which resulted in a lengthy comma-separated
list of integers.
This change standardizes serialization, making it more efficient and
reducing the size of the responses. The previous format, with a series
of comma-separated integers for pubkey-like fields, looked odd.
Then:
```
curl -X POST\
-H "Content-Type: application/json" \
--data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [491882, "sgx"] }' \
https://mainnet.era.zksync.io
{"jsonrpc":"2.0","result":[{"attestation":[3,0,2,0,0,0,0,0,10,<dozens of comma-separated integers here>
```
Now:
```
$ curl -X POST \
-H "Content-Type: application/json" \
--data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [1, "sgx"] }' \
http://localhost:3050
{"jsonrpc":"2.0","result":[{"l1BatchNumber":1,"teeType":"sgx","pubkey":"0506070809","signature":"0001020304","proof":"0a0b0c0d0e","provedAt":"2024-09-16T11:53:38.253033Z","attestation":"0403020100"}],"id":1}
```
This change needs to be deployed in lockstep with:
https://github.com/matter-labs/teepot/pull/209.
## Why ❔
To improve user experience (UX) and optimize the RPC response size.
## Checklist
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
Co-authored-by: Artem Makhortov <13339874+artmakh@users.noreply.github.com>
* feat(ci): Add runtime-base image (#2919)
## What ❔
Adding image which will be used as runtime-base for services
## Why ❔
As a workaround for dockerhub ratelimits
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* fix(ci): Rename workflows for runner and builder base (#2921)
## What ❔
Rename workflows for runner and builder base
## Why ❔
Easier to find =)
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* fix(ci): zksync-runtime-base proper platforms names (#2923)
## What ❔
Proper platform names for docker build action
## Why ❔
Docker documentation (which is linked in docker-build-action docs)
[points to non-working
reference](https://docs.docker.com/reference/cli/docker/buildx/build/#platform)
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* fix(ci): Proper type of var for platforms in docker build action (#2924)
## What ❔
Proper type of var for platforms in docker build action
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* fix(zk_toolbox): show readable log (#2911)
## What ❔
<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
---------
Signed-off-by: Danil <deniallugo@gmail.com>
* feat(ci): Migrate to vendored image as a zksync-runtime-base (#2926)
## What ❔
Move building images to zksync-runtime-base
## Why ❔
Workaround for dockerhub rate limits
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* chore(main): release core 24.25.0 (#2816)
:robot: I have created a release *beep* *boop*
---
##
[24.25.0](https://github.com/matter-labs/zksync-era/compare/core-v24.24.0...core-v24.25.0)
(2024-09-19)
### Features
* (DB migration) Rename recursion_scheduler_level_vk_hash to
snark_wrapper_vk_hash
([#2809](https://github.com/matter-labs/zksync-era/issues/2809))
([64f9551](https://github.com/matter-labs/zksync-era/commit/64f95514c99f95da2a19a97ff064c29a97efc22f))
* add da clients
([#2743](https://github.com/matter-labs/zksync-era/issues/2743))
([9218612](https://github.com/matter-labs/zksync-era/commit/9218612fdb2b63c20841e2e2e5a45bbd23c01fbc))
* attester committees data extractor (BFT-434)
([#2684](https://github.com/matter-labs/zksync-era/issues/2684))
([92dde03](https://github.com/matter-labs/zksync-era/commit/92dde039ee8a0bc08e2019b7fa6f243a34d9816f))
* emit errors in prover API metrics
([#2890](https://github.com/matter-labs/zksync-era/issues/2890))
([2ac7cc5](https://github.com/matter-labs/zksync-era/commit/2ac7cc5836e69fc82c98df2005fedee01c1084e1))
* **en:** Resume incomplete snapshot in snapshot creator in more cases
([#2886](https://github.com/matter-labs/zksync-era/issues/2886))
([f095b4a](https://github.com/matter-labs/zksync-era/commit/f095b4a3223222ac712de53592fe1e68f766600f))
* make `to` address optional for transaction data
([#2852](https://github.com/matter-labs/zksync-era/issues/2852))
([8363c1d](https://github.com/matter-labs/zksync-era/commit/8363c1d8697ad9bd2fe5d326218476bc3dad38af))
* **prover:** Optimize setup keys loading
([#2847](https://github.com/matter-labs/zksync-era/issues/2847))
([19887ef](https://github.com/matter-labs/zksync-era/commit/19887ef21a8bbd26977353f8ee277b711850dfd2))
* Selector generator tool
([#2844](https://github.com/matter-labs/zksync-era/issues/2844))
([b359b08](https://github.com/matter-labs/zksync-era/commit/b359b085895da6582f1d28722107bc5b25f1232c))
* **tee:** use hex serialization for RPC responses
([#2887](https://github.com/matter-labs/zksync-era/issues/2887))
([abe0440](https://github.com/matter-labs/zksync-era/commit/abe0440811ae4daf4a0f307922a282e9664308e0))
* **utils:** Rework locate_workspace, introduce Workspace type
([#2830](https://github.com/matter-labs/zksync-era/issues/2830))
([d256092](https://github.com/matter-labs/zksync-era/commit/d2560928cc67b40a97a5497ac8542915bf6f91a9))
* **zk_toolbox:** Add external_node consensus support
([#2821](https://github.com/matter-labs/zksync-era/issues/2821))
([4a10d7d](https://github.com/matter-labs/zksync-era/commit/4a10d7d9554d6c1aa2f4fc46557d40baaad8ff2f))
### Bug Fixes
* count SECP256 precompile to account validation gas limit as well
([#2859](https://github.com/matter-labs/zksync-era/issues/2859))
([fee0c2a](https://github.com/matter-labs/zksync-era/commit/fee0c2ad08a5ab4a04252765b367eb9fbb1f3db7))
* **en:** Fix connection starvation during snapshot recovery
([#2836](https://github.com/matter-labs/zksync-era/issues/2836))
([52f4f76](https://github.com/matter-labs/zksync-era/commit/52f4f763674d25f8a5e7f3a111354a559f798d52))
* **eth_watch:** fix `get_events_inner`
([#2882](https://github.com/matter-labs/zksync-era/issues/2882))
([c957dd8](https://github.com/matter-labs/zksync-era/commit/c957dd8011213e0e95fa5962e2310321b29a0d16))
* handling of HTTP 403 thrown by proxyd
([#2835](https://github.com/matter-labs/zksync-era/issues/2835))
([2d71c74](https://github.com/matter-labs/zksync-era/commit/2d71c7408a0eed3662fc51f70fa9f525d66e4c6f))
* **state-keeper:** Restore processed tx metrics in state keeper
([#2815](https://github.com/matter-labs/zksync-era/issues/2815))
([4d8862b](https://github.com/matter-labs/zksync-era/commit/4d8862b76a55ac78edd481694fefd2107736ffd9))
* **tee-prover:** fix deserialization of `std::time::Duration` in `envy`
config ([#2817](https://github.com/matter-labs/zksync-era/issues/2817))
([df8641a](https://github.com/matter-labs/zksync-era/commit/df8641a912a8d480ceecff58b0bfaef05e04f0c8))
---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
---------
Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>
* docs: Update system requirements (#2894)
## What ❔
- 30 GB of free disk space are no longer sufficient to run a testnet EN.
Upped the system requirements to 50 GB.
- State growth is normally used to refer to the growth of the state
tree, not of the historical state. What we have is historical state
pruning, not state pruning.
* chore(vm): Bump vm2 repo revision (#2883)
## What ❔
Bumps the `vm2` crate revision and updates new VM usage correspondingly.
## Why ❔
To keep the `vm2` dependency up to date and track possible regressions.
## Checklist
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
* feat(nix): use precompiled `rocksdb` and `snappy` (#2925)
## What ❔
Use precompiled `rocksdb` and `snappy` in the nix build.
## Why ❔
Speeds up compilation and is usually better maintained.
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
Signed-off-by: Harald Hoyer <harald@matterlabs.dev>
* feat(prover): Add endpoint to PJM to get queue reports (#2918)
## What ❔
Add `/queue_report` endpoint, which will get the data about queue and
send it.
## Why ❔
To work with new autoscaler
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
---------
Co-authored-by: EmilLuta <EmilLuta@users.noreply.github.com>
* fix(eth-sender): print better error message in case of missing blob prices (#2927)
Signed-off-by: tomg10 <lemures64@gmail.com>
Co-authored-by: perekopskiy <mikeson.dp@gmail.com>
* feat: added seed_peers to consensus global config (#2920)
It will allow us to announce the recommended default list of peers to
all ENs without manual intervention.
Fixes BFT-509.
---------
Co-authored-by: Bruno França <bruno@franca.xyz>
* feat(vm): Do not panic on VM divergence (#2705)
## What ❔
- Allows to continue batch execution on divergence via new
`ShadowLenient` VM mode.
- Dumps VM state to logs and optionally a file on divergence.
## Why ❔
Allows to detect divergencies in multiple batches w/o blockers. The
dumped VM state will hopefully allow investigating divergencies locally,
although this logic isn't implemented yet.
## Checklist
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
* feat(prover): Use query macro instead string literals for queries (#2930)
## What ❔
In some places our sqlx queries are using string literals instead of
query macros. This PR changes this behaviour in places it is possible.
## Why ❔
To prevent possible SQL injections.
It also will cache the queries, which should make them faster.
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* feat(circuit_prover): Add circuit prover (#2908)
### Summary
This PR introduces a new component `circuit_prover`, which is multiple
WVGs & GPU prover running together, groupless. The changes are
non-destructive, old setup and new setup must work together in-tandem.
### What?
Circuit prover is a component that runs X WVGs alongside a GPU prover.
Making full use of CPUs on the GPU machine, WVGs as a component can be
removed altogether. Prover groups are not needed anymore. Based on
empiric testing we can (almost -- there will be follow-up fixes to make
it efficient) fully run everything on a single machine.
The current implementation can sunset the old setup. Current metrics
show that circuit prover is > 60% efficient than old one (but quirks are
needed for node proofs to unlock it -- will be treated as a follow-up).
The purpose is to have the `circuit_prover` deprecate old `prover_fri` &
`witness_vector_generator`.
### Why?
The changes will allow us to reduce our infrastructure footprint by ~2x
and fix plenty of issues we had in the past. Namely:
- fully decoupled of GCP
- better resource utilization & reduce costs
- reduce overall infrastructure needs (which solves the GPU
unavailability we've been facing)
- reduce complexity & other inefficiencies (no more prover groups!)
- and more
### Ask
We want to unblock folks running on AWS. This PR is done as is to speed
up release process on DevOps side, as it's the longest pole.
NOTE: This is the first PR out of a longer set of PRs.
Comments are more than welcome, but the following concerns will be
addressed in follow-up PRs and are out of scope for this PR:
- tracing implementation is subpar; in fact, I'm confident that most
metrics could be done via traces
- there's a lot of code duplication (both old/new prover, but runner
interface between new WVG & CP)
- tests
- concern separation between job scheduling & job execution
- job priority based on resource consumption
- other nits (such as no README, constants being hard-coded instead of
configuration, etc.)
### Reviewer suggestion
This is basically a merge between `prover_fri`,
`witness_vector_generation` and `JobProcessor`. Checking code alongside
should give you a better view of what's going on. Sorry for making this
hard. :/
* refactor(vm): Refactor L1 batch params provider (#2891)
## What ❔
Refactors `L1BatchParamsProvider`, in particular its construction.
## Why ❔
To have more intuitive DevEx.
## Checklist
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
* feat(vm): Extract oneshot VM executor – environment types (#2885)
## What ❔
Continues oneshot VM executor extraction by moving generic environment
types to the `zksync_vm_executor` crate.
## Why ❔
Makes it possible to use the VM executor outside the API server. E.g.,
it's now used in `zksync_node_consensus`.
## Checklist
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
* fix(api_server): fix blob_gas length (#2673)
Discovered during testing Gateway integration (our own asserts from
query.rs started being triggered by our eth api implementation :D )
---------
Co-authored-by: perekopskiy <mikeson.dp@gmail.com>
* refactor: Improve EN Grafana dashboards (#2931)
## What ❔
Adding more information to the EN Grafana dashboards, which barely had
any.
## Why ❔
Better observability.
Fixes BFT-507
* chore(main): release core 24.26.0 (#2928)
:robot: I have created a release *beep* *boop*
---
##
[24.26.0](https://github.com/matter-labs/zksync-era/compare/core-v24.25.0...core-v24.26.0)
(2024-09-23)
### Features
* added seed_peers to consensus global config
([#2920](https://github.com/matter-labs/zksync-era/issues/2920))
([e9d1d90](https://github.com/matter-labs/zksync-era/commit/e9d1d905f1ce86f9de2cf39d79be4b5aada4a81d))
* **circuit_prover:** Add circuit prover
([#2908](https://github.com/matter-labs/zksync-era/issues/2908))
([48317e6](https://github.com/matter-labs/zksync-era/commit/48317e640a00b016bf7bf782cc94fccaf077ed6d))
* **prover:** Add endpoint to PJM to get queue reports
([#2918](https://github.com/matter-labs/zksync-era/issues/2918))
([2cec83f](https://github.com/matter-labs/zksync-era/commit/2cec83f26e0b9309387135ca43718af4fcd6f6b1))
* **vm:** Do not panic on VM divergence
([#2705](https://github.com/matter-labs/zksync-era/issues/2705))
([7aa5721](https://github.com/matter-labs/zksync-era/commit/7aa5721d22e253d05d369a60d5bcacbf52021c48))
* **vm:** Extract oneshot VM executor – environment types
([#2885](https://github.com/matter-labs/zksync-era/issues/2885))
([a2d4126](https://github.com/matter-labs/zksync-era/commit/a2d4126f9e0c9dc46f49a861549d076fdbcf66d3))
### Bug Fixes
* **api_server:** fix blob_gas length
([#2673](https://github.com/matter-labs/zksync-era/issues/2673))
([44a8f79](https://github.com/matter-labs/zksync-era/commit/44a8f79739704e1216a7b34a0580ad7ba1cce3bd))
* **eth-sender:** print better error message in case of missing blob
prices ([#2927](https://github.com/matter-labs/zksync-era/issues/2927))
([38fc824](https://github.com/matter-labs/zksync-era/commit/38fc824f75e8b0e84f10348d1502fc8a26d12015))
---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
---------
Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>
* chore(ci): Fix CI building (#2936)
After new prover was introduced, a new flag is needed for code that
touches GPU related code that has no CUDA available. This PR adds the
flag to CI pipelines that were missed.
[Sample](https://github.com/matter-labs/zksync-era/actions/runs/10958933706/job/30430256489)
broken CI.
* feat(vm): Split old and new VM implementations (#2915)
## What ❔
Splits old and new VM implementations in the `multivm` crate:
- Old VMs are encapsulated in the `LegacyVmInstance` enum, while new
ones in the `FastVmInstance` enum (which includes plain and shadowed VM
variants).
- Fast VM and `FastVmInstance` now expose a tracer type.
- Usage of the Fast VM in the batch executor are updated
correspondingly.
## Why ❔
It seems infeasible to unify the tracer model for old and new VMs, so
keeping them all in a single enum makes little sense.
## Checklist
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
* feat(zk_toolbox): Add options for selective installation of zk_inception and zk_supervisor (#2934)
## What ❔
- Introduced `--inception` and `--supervisor` options to allow users to
specify which binaries to install.
- Removed the `ZKUP_SKIP_ZK_SUPERVISOR` option.
## Why ❔
Make it clear what to install, focusing on what to include rather what
to exclude
* feat(zk_toolbox): Add zk toolbox unit tests to zks test (#2935)
## What ❔
Add zk toolbox unit tests to zks test
* feat(circuit-prover): Add circuit_prover Dockerfile and build rules (#2939)
This will enable us to build images for circuit proving (new version).
Not tested.
---------
Co-authored-by: Yury Akudovich <ya@matterlabs.dev>
* feat(contract-verifier): add new compilers (#2947)
* ci: Copy setup_keys to docker/circuit-prover-gpu-gar (#2951)
## What ❔
Copy setup keys to be available for docker/circuit-prover-gpu-gar.
<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->
## Why ❔
To build circuit-prover-gpu image with keys.
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
* docs: fix invalid links to "namespaces/eth.rs" and "execution_sandbox.rs" (#2945)
## What ❔
This PR fixes 2 invalid links in the documents which was caused by
refactor (refactor: Split the rest of the zksync_core
(https://github.com/matter-labs/zksync-era/pull/1940))
## Why ❔
Unclarity in documentation
## Checklist
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* chore(prover): Fixes broken non-GPU setup (#2953)
Previously, this was done via a compile flag. This was deemed poor taste
and we've moved a specific env-var - `ZKSYNC_USE_CUDA_STUBS`. This is
wired deep within crypto dependencies.
This PR updates the dependencies, adds documentation on how to work with
the flag and updates all workflows (fixing the broken ones from
RUSTFLAGS time).
---------
Co-authored-by: Igor Aleksanov <popzxc@yandex.ru>
* chore(prover-gateway): Speed up polling (#2957)
16 minutes is just ridiculous. This should've been done a long time ago,
but has been avoided due to other priorities and always put on the
back-burner/forgotten. Today we change it.
* fix(api): Return correct flat call tracer (#2917)
## What ❔
It was an inconsistency between call tracer results
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
---------
Signed-off-by: Danil <deniallugo@gmail.com>
* chore(main): release core 24.27.0 (#2940)
:robot: I have created a release *beep* *boop*
---
##
[24.27.0](https://github.com/matter-labs/zksync-era/compare/core-v24.26.0...core-v24.27.0)
(2024-09-25)
### Features
* **vm:** Split old and new VM implementations
([#2915](https://github.com/matter-labs/zksync-era/issues/2915))
([93bc66f](https://github.com/matter-labs/zksync-era/commit/93bc66f21f9f67a440f06f1c4402e0d687698741))
### Bug Fixes
* **api:** Return correct flat call tracer
([#2917](https://github.com/matter-labs/zksync-era/issues/2917))
([218646a](https://github.com/matter-labs/zksync-era/commit/218646aa1c56200f4ffee99b7f83366e2689354f))
---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
---------
Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>
* feat(eth-watch): redesign to support multiple chains (#2867)
This PR contains code from sync-layer-stable + I removed old upgrades
processor and updated unit-tests to use the new one
---------
Signed-off-by: tomg10 <lemures64@gmail.com>
* feat(prover): WG refactoring #3 (#2942)
## What ❔
Create WG struct with generic round param.
Implement generic JobProcessor for all types of WG.
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* chore(eth-sender): remove deprecated proof loading (#2958)
## What ❔
Removes deprecated proof loading for eth-sender
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* fix(eth-watch): add missing check that from_block is not larger than finalized_block (#2969)
Signed-off-by: tomg10 <lemures64@gmail.com>
* refactor: Deleted unnecessary files (#2959)
## What ❔
- ~`.cargo/config.toml` is a remnant from when we used private repos. It
should no longer be necessary.~
- ~`configs/` doesn't seem to be used at all.~
- `building-from-scratch` just contains an old docker file that was used
for Coinbase. No longer used.
* fix: cargo-deny advisory due to structopt (#2972)
## What ❔
Adds https://rustsec.org/advisories/RUSTSEC-2024-0375 to `deny.toml`,
which is due to `StructOpt`.
<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->
## Why ❔
* Currently it's impossible to migrate to clap as
[zkevm_test_harness](https://github.com/matter-labs/era-zkevm_test_harness/blob/v1.5.0/Cargo.toml#L46)
still depends on structopt.
* This
[unblocks](https://github.com/matter-labs/zksync-era/actions/runs/11054218994/job/30710479095)
the CI
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
* fix: ignore unknown fields in rpc json response (#2962)
In some previous pr, I've added a field to json RPC response
(consensusGlobalConfig) and it turned out to be incompatible due to the
default parsing behavior: an error was returned when unknown fields were
present. In general, for programmatic communications the recommended
behaviour is to silently ignore unknown fields, so that adding new
fields is compatible.
I've also weakened the restriction on accepting new consensus genesis by
external nodes - when the unknown fields are silently dropped, it is
possible that the node will perceive genesis with the same fork number
differently after a binary update (because it receives more fields).
* fix(zk_toolbox): Do not panic if mint is not successful (#2973)
## What ❔
Use governor for minting process and do not panic if minting is not
successful
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
Signed-off-by: Danil <deniallugo@gmail.com>
* initial commit
* feat: zk_toolbox command for setting attester committee (#2900)
zk_toolbox command for setting attester committee in consensus registry
contract.
Fixes BFT-505
---------
Co-authored-by: Bruno França <bruno@franca.xyz>
* fix: chainstack block limit exceeded (#2974)
## What ❔
Added support for chainstack developer and growth plan.
## Why ❔
I was getting the below error with chainstack's growth plan when I tried
run self deployed zksync hyperchain. To fix this, I have added
appropriate changes to the code.
```
ERROR zksync_eth_watch: Failed to process new blocks: Eth client error: JSON-RPC request get_logs(filter=Filter { from_block: Some(Number(6732838)), to_block: Some(Number(6757429)), block_hash: None, address: Some(ValueOrArray([0xadd82b0c015be3fb7e8410061b4f15d577b4207f, 0xd6ee28b24f87c1d634540699b00e9c37fdc7b3bd, 0x41b75786db38a486c99972c8322fcb14c4f000a2, 0x5d300908338b85bee3f30f2e0ad196fc1587b70d])), topics: Some([Some(ValueOrArray([0x4531cd5795773d7101c17bdeb9f5ab7f47d7056017506f937083be5d6e77a382, 0x23bc9f5dc037eb49c162fd08c2a4d43dfe70063149e140d502273168da0a0625, 0xd50ef21701c8ef211433b140724b8d6de471e7d822c8a616c3d424fe2d0e98a9]))]), limit: None }) failed: ErrorObject { code: InvalidParams, message: "Block range limit exceeded. See more details at https://docs.chainstack.com/docs/limits#evm-range-limits", data: None }
```
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
* feat(eth-sender): add a cap to time_in_mempool (#2978)
Signed-off-by: tomg10 <lemures64@gmail.com>
* chore: update derive_more to stable version (#2961)
## What ❔
Updates `derive_more` to a stable version.
<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->
## Why ❔
We're currently on beta and this is a requisite for supporting
[foundry-zksync](https://github.com/matter-labs/foundry-zksync).
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
* use Notify for a more deterministic approach
* feat(da-clients): add secrets (#2954)
## What ❔
Add an opportunity to store the sensitive information of DA clients in
the secrets.
## Why ❔
Some sensitive info is currently stored as plaintext, which is not
secure, we should use secrets for those values
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
---------
Co-authored-by: Danil <deniallugo@gmail.com>
* fix(api): Fix batch fee input for `debug` namespace (#2948)
## What ❔
Changes the `debug_traceCall` handler to use the fee provider for recent
blocks (pending / latest / committed).
## Why ❔
Previously, the `debug` namespace used a static batch fee input
retrieved at the node initialization. This looks incorrect.
## Checklist
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
* feat(zk_toolbox): add fees integration test to toolbox (#2898)
## What ❔
<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->
Add fees integration test to toolbox and CI.
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
---------
Signed-off-by: Danil <deniallugo@gmail.com>
Co-authored-by: Danil <deniallugo@gmail.com>
* feat: Expose http debug page (#2952)
## What ❔
Exposes the consensus http debug on port 5000.
Fixes BFT-508
* feat(zk_inception): Add ecosystem ports scanner (#2849)
## What ❔
Added `EcosystemPortsScanner` tool to scan ecosystem directory config
and find all values for ports.
Returns `EcosystemPorts` object with ports found and custom description
of a service.
`EcosystemPorts` also contains methods to: check whether the port is
assigned, allocate a new port in range, etc.
## Why ❔
With multiple chains and number of services (like explorer, etc.)
growing - it is useful to have a tool (`EcosystemPorts`) that can be
used to allocate new ports, thus, lowering the chance of port conflicts
within created configuration.
## Checklist
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
---------
Co-authored-by: Matías Ignacio González <maigonzalez@fi.uba.ar>
* build: Set consensus and pruning as default for EN (#2956)
## What ❔
ENs, started from docker compose, now have consensus and pruning enabled
as default. Also, the EN docs were updated.
* fix(ci): Remove deprecated artifacts dir in contracts (#2989)
## What ❔
Removing deprecated `artifacts` dir from build process
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* chore(zk_toolbox): Use load current chain function (#2990)
## What ❔
<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
Signed-off-by: Danil <deniallugo@gmail.com>
* perf(api): More efficient gas estimation (#2937)
## What ❔
Makes gas estimation more efficient by using heuristics similar to ones
used by Ethereum clients:
- Defines the lower gas limit bound as the gas consumed with an
effectively infinite gas limit.
- Defines an initial binary search pivot as a slightly larger value so
that in the best / average case, most of the search space is immediately
discarded.
## Why ❔
Improves gas estimation latency in the best and average case.
## Checklist
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
* test: Base token price client, fix: ratio calculation multiplicative inverse error (#2970)
## What ❔
- I introduce tests to external_price_api crate. Some work was taken
from https://github.com/matter-labs/zksync-era/pull/2315
- fix multiplicative inverse ratio calculation bug
- fix missing checks in get_fraction
## Why ❔
- We want to make sure the flow of data from API response to token ratio
is correct
## Checklist
- [X] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [X] Tests for the changes have been added / updated.
- [X] Documentation comments have been added / updated.
- [X] Code has been formatted via `zk fmt` and `zk lint`.
* chore(githooks): Use zk_toolbox in githooks (#2895)
## What ❔
Switch to zk_supervisor fmt in our githooks
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
---------
Signed-off-by: Danil <deniallugo@gmail.com>
* feat(zk_toolbox): Add SQL format for zk supervisor (#2950)
## What ❔
Add SQL format for zk supervisor
* feat(ci): Add Makefile for easy local build (#2985)
## What ❔
Add Makefile for easier local build of all components
## Why ❔
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* fix(zk_toolbox): wrong configs path for prover binaries (#2922)
## What ❔
Right now, while running prover binaries inside of zksync-era, config
paths are specified as path to config from chain config(which is
`./chain/configs` - relative path, but not absolute). This breaks,
because to run prover binaries, shell enters prover directory of era.
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* feat(zk_toolbox): Migrate db during update (#2995)
## What ❔
Update databases during update process, now it's a universal receipt how
to pull changes
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor
lint`.
Signed-off-by: Danil <deniallugo@gmail.com>
* fix(ci): Bring back hack for contracts build till full migration to foundry (#3000)
## What ❔
Creation of empty dirs, to support building from both new
(foundry-built) and old contracts
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor
lint`.
* chore(main): release core 24.28.0 (#2968)
:robot: I have created a release *beep* *boop*
---
##
[24.28.0](https://github.com/matter-labs/zksync-era/compare/core-v24.27.0...core-v24.28.0)
(2024-10-02)
### Features
* **da-clients:** add secrets
([#2954](https://github.com/matter-labs/zksync-era/issues/2954))
([f4631e4](https://github.com/matter-labs/zksync-era/commit/f4631e4466de620cc1401b326d864cdb8b48a05d))
* **eth-sender:** add a cap to time_in_mempool
([#2978](https://github.com/matter-labs/zksync-era/issues/2978))
([650d42f](https://github.com/matter-labs/zksync-era/commit/650d42fea6124d80b60a8270a303d72ad6ac741e))
* **eth-watch:** redesign to support multiple chains
([#2867](https://github.com/matter-labs/zksync-era/issues/2867))
([aa72d84](https://github.com/matter-labs/zksync-era/commit/aa72d849c24a664acd083eba73795ddc5d31d55f))
* Expose http debug page
([#2952](https://github.com/matter-labs/zksync-era/issues/2952))
([e0b6488](https://github.com/matter-labs/zksync-era/commit/e0b64888aae7324aec2d40fa0cd51ea7e1450cd9))
* **zk_toolbox:** add fees integration test to toolbox
([#2898](https://github.com/matter-labs/zksync-era/issues/2898))
([e7ead76](https://github.com/matter-labs/zksync-era/commit/e7ead760ce0417dd36af3839ac557f7e9ab238a4))
* **zk_toolbox:** Add SQL format for zk supervisor
([#2950](https://github.com/matter-labs/zksync-era/issues/2950))
([540e5d7](https://github.com/matter-labs/zksync-era/commit/540e5d7554f54e80d52f1bfae37e03ca8f787baf))
### Bug Fixes
* **api:** Fix batch fee input for `debug` namespace
([#2948](https://github.com/matter-labs/zksync-era/issues/2948))
([79b6fcf](https://github.com/matter-labs/zksync-era/commit/79b6fcf8b5d10a0ccdceb846370dd6870b6a32b5))
* chainstack block limit exceeded
([#2974](https://github.com/matter-labs/zksync-era/issues/2974))
([4ffbf42](https://github.com/matter-labs/zksync-era/commit/4ffbf426de166c11aaf5d7b5ed7d199644fba229))
* **eth-watch:** add missing check that from_block is not larger than
finalized_block
([#2969](https://github.com/matter-labs/zksync-era/issues/2969))
([3f406c7](https://github.com/matter-labs/zksync-era/commit/3f406c7d0c0e76d798c2d838abde57ca692822c0))
* ignore unknown fields in rpc json response
([#2962](https://github.com/matter-labs/zksync-era/issues/2962))
([692ea73](https://github.com/matter-labs/zksync-era/commit/692ea73f75a5fb9db2b4ac33ad24d20568638742))
### Performance Improvements
* **api:** More efficient gas estimation
([#2937](https://github.com/matter-labs/zksync-era/issues/2937))
([3b69e37](https://github.com/matter-labs/zksync-era/commit/3b69e37e470dab859a55787f6cc971e7083de2fd))
---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
---------
Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>
* feat(zk_toolbox): Add subcommands and flags for chain registration (#2946)
## What ❔
Added subcommands:
`zki chain init configs` - just creates configs with an intention to run
chain initialization manually via subcommands
`zki chain register-chain` - runs steps from `RegisterHyperchain.s.sol`
`zki chain accept-chain-ownership` - accepts ownership for
`DiamondProxy`
`zki chain genesis database` - initializes database only, performs
migration (uses values from args or `secrets.yaml`)
`zki chain genesis server` - runs server --genesis
Added flags:
`zki ecosystem init --ecosystem-only` - runs `init` only for ecosystem
(skips `init` for chain)
Other changes:
* Fixed issue with `--wallet_path` value ignored
* Nullify database names if `zki ecosystem init` is used for multiple
chains
* Zeroify some addresses in `contracts.yaml` when copying from ecosystem
during init
## Why ❔
These changes allow us to run the chain registration process for
externally hosted chains.
Not ideal yet, but the process goes like this:
L1 side:
* Init ecosystem: `zki ecosystem create && zki ecosystem init
--ecosystem-only && zki chain init configs`
* Fill in wallets
* Register chain: `zki chain register-chain`
* Deploy L2 contracts: `zki chain deploy-l2-contracts`
* Share `contracts.yaml`
L2 side:
* Init ecosystem: `zki ecosystem create && zki ecosystem init
--ecosystem-only && zki chain init configs`
* Fill in wallets
* Copy `contracts.yaml`
* Accept ownership: `zki chain accept-chain-ownership`
* Initialize databases: `zki chain genesis database`
* Run server genesis: `zki chain genesis server`
## Checklist
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
* fix(zk_toolbox): Correct secrets (#3004)
## What ❔
<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor
lint`.
Signed-off-by: Danil <deniallugo@gmail.com>
* replace atomic for mutex
* move const to config
* feat(ci): Add external-node build to Makefile (#3002)
## What ❔
Add external-node build to Makefile
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor
lint`.
* fix(zk_toolbox): Remove prover db from server init (#3009)
## What ❔
Remove prover db from server init
## Why ❔
prover db is not needed in server init since prover db is initialized
with prover init
* fix(ci): Build zk-env with CUDA (#3013)
## What ❔
Ability to build zk-env with cuda not only from push to main branch
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor
lint`.
* Update contracts
* Finalize merge
* feat(en): periodically fetch bridge addresses (#2949)
## What ❔
Periodically fetch bridge addresses
## Why ❔
Addresses will be changed during gateway upgrade
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
* feat(eth-watch): catch another reth error (#3026)
## What ❔
<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor
lint`.
* test(api): Add more `TxSender` tests (#3001)
## What ❔
Covers more `TxSender` functionality with unit tests.
## Why ❔
More test coverage is good. Also, it will help with new VM integration
(by running the added tests with a shadowed VM).
## Checklist
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor
lint`.
* feat(eth-sender): add time_in_mempool_cap config (#3018)
Configuration parameter for time_in_mempool_cap, cap for time_in_mempool
in eth-sender fee model + default values for parameter_a and parameter_b
for GasAdjuster
* fix(zkstack_cli): Fix fmt --check (#3027)
## What ❔
`fmt --check` wasn't returning an error code when failing
* docs(zk_toolbox): remove generate-sk subcommand references (#3016)
## What ❔
<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->
Remove `generate-sk` subcommand which doesn’t exist anymore.
## Why ❔
<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->
## Checklist
<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor
lint`.
* chore(vm): Remove tests for old VM versions (#3015)
## What ❔
Removes unit test files for old VM versions in the `multivm` crate.
## Why ❔
These tests are not built (not included into the module tree of the
crate), so they are just deadweight.
## Checklist
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
* Update contracts
* refactor(eth): Brush up `eth_signer` crate (#3014)
## What ❔
Brushes up the `eth_signer` crate so that its API is easier to use
(e.g., it doesn't require an async runtime).
## Why ❔
Improved DevEx.
## Checklist
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor
lint`.
* Update query jsons
* fix(vm): Prepare new VM for use in API server and fix divergences (#2994)
## What ❔
Prepares the new VM for use in the API server (without tracers):
- Fills in more statistics
- Covers more statistic fields in divergence checks and fixes the
corresponding divergence in `pubdata_published`
- Fixes a divergence if revert reaches the bootloader call frame
(happens in the call execution mode)
- Fixes a panic in `ShadowVm` if the transaction hash is not set and
makes `impl Debug for Transaction` non-panicking. (A hash is not set if
a transaction is converted from a call.)
## Why ❔
Part of preparations for integrating the new VM into the API server.
## Checklist
- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
---------
Co-authored-by: Joonatan Saarhelo <joon.saar@gmail.com>
* feat: Increased the timeout of waiting for db in tests (#3007)
I don't know about other people's setup, but for me it is very flaky
with just 3s
* feat: add metric to track current cbt ratio (#3020)
Add a new metric to track custom base token to ETH ratio
* Format md
* Refactor secrets
---------
Signed-off-by: Danil <deniallugo@gmail.com>
Signed-off-by: Harald Hoyer <harald@matterlabs.dev>
Signed-off-by: tomg10 <lemures64@gmail.com>
Co-authored-by: Artem Makhortov <13339874+artmakh@users.noreply.github.com>
Co-authored-by: Alex Ostrovski <slowli@users.noreply.github.com>
Co-authored-by: Danil <deniallugo@gmail.com>
Co-authored-by: Nacho Avecilla <ignacio.avecilla@lambdaclass.com>
…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What ❔
Continues oneshot VM executor extraction by moving generic environment types to the
zksync_vm_executorcrate.Why ❔
Makes it possible to use the VM executor outside the API server. E.g., it's now used in
zksync_node_consensus.Checklist
zk fmtandzk lint.