feat(prover): parallelized memory queues simulation in BWG #2652
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 tasks
|
Also requires an update in shivini |
EmilLuta
reviewed
Aug 15, 2024
There was a problem hiding this comment.
I spent some time going through this. Uh, so overall, I get the main idea. That said, a lot of complexity is moved from harness to BWG and that I don't like. I understand the split that's happening now, but I don't feel very comfortable maintaining/changing/troubleshooting the circuit recombining part. Do you envision any other method where less harness magic is spilled into prover subsystem?
So I do not see now any other way to deal with it other then split monolithic ram permutaion circuit artifacts |
|
Maybe we can come up with a more elegant way to store/recombine data? |
EmilLuta
approved these changes
Aug 16, 2024
This was referenced Aug 16, 2024
github-merge-queue bot
pushed a commit
that referenced
this pull request
Aug 21, 2024
🤖 I have created a release *beep* *boop* --- ## [24.19.0](core-v24.18.0...core-v24.19.0) (2024-08-21) ### Features * **db:** Allow creating owned Postgres connections ([#2654](#2654)) ([47a082b](47a082b)) * **eth-sender:** add option to pause aggregator for gateway migration ([#2644](#2644)) ([56d8ee8](56d8ee8)) * **eth-sender:** added chain_id column to eth_txs + support for gateway in tx_aggregator ([#2685](#2685)) ([97aa6fb](97aa6fb)) * **eth-sender:** gateway support for eth tx manager ([#2593](#2593)) ([25aff59](25aff59)) * **prover_cli:** Add test for status, l1 and config commands. ([#2263](#2263)) ([6a2e3b0](6a2e3b0)) * **prover_cli:** Stuck status ([#2441](#2441)) ([232a817](232a817)) * **prover:** Add ProverJobMonitor ([#2666](#2666)) ([e22cfb6](e22cfb6)) * **prover:** parallelized memory queues simulation in BWG ([#2652](#2652)) ([b4ffcd2](b4ffcd2)) * update base token rate on L1 ([#2589](#2589)) ([f84aaaf](f84aaaf)) * **zk_toolbox:** Add zk_supervisor run unit tests command ([#2610](#2610)) ([fa866cd](fa866cd)) * **zk_toolbox:** Run formatters and linterrs ([#2675](#2675)) ([caedd1c](caedd1c)) ### Bug Fixes * **contract-verifier:** Check for 0x in zkvyper output ([#2693](#2693)) ([0d77588](0d77588)) * make set token multiplier optional ([#2696](#2696)) ([16dff4f](16dff4f)) * **prover:** change bucket for RAM permutation witnesses ([#2672](#2672)) ([8b4cbf4](8b4cbf4)) * use lower fair l2 gas price for cbt ([#2690](#2690)) ([e1146fc](e1146fc)) ### Performance Improvements * **logs-bloom:** do not run heavy query if migration was completed ([#2680](#2680)) ([f9ef00e](f9ef00e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Aug 28, 2024
🤖 I have created a release *beep* *boop* --- ## [16.5.0](prover-v16.4.0...prover-v16.5.0) (2024-08-28) ### Features * **prover_cli:** Add test for status, l1 and config commands. ([#2263](#2263)) ([6a2e3b0](6a2e3b0)) * **prover_cli:** Stuck status ([#2441](#2441)) ([232a817](232a817)) * **prover:** Add ProverJobMonitor ([#2666](#2666)) ([e22cfb6](e22cfb6)) * **prover:** parallelized memory queues simulation in BWG ([#2652](#2652)) ([b4ffcd2](b4ffcd2)) * Provide easy prover setup ([#2683](#2683)) ([30edda4](30edda4)) ### Bug Fixes * **prover_cli:** Remove congif file check ([#2695](#2695)) ([2f456f0](2f456f0)) * **prover_cli:** Update prover cli README ([#2700](#2700)) ([5a9bbb3](5a9bbb3)) * **prover:** change bucket for RAM permutation witnesses ([#2672](#2672)) ([8b4cbf4](8b4cbf4)) * **prover:** fail when fri prover job is not found ([#2711](#2711)) ([8776875](8776875)) * **prover:** Revert use of spawn_blocking in LWG/NWG ([#2682](#2682)) ([edfcc7d](edfcc7d)) * **prover:** speed up LWG and NWG ([#2661](#2661)) ([6243399](6243399)) * **vm:** Fix used bytecodes divergence ([#2741](#2741)) ([923e33e](923e33e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What ❔
The updated logic of memory queue simulation in the test_harness requires changing the structure of the BWG artifacts. When processing memory queues for ram permutation circuit, part of the witness is sent to storage separately. For this reason, a new type of circuit wrapper has been added - partial base circuit. Also added logic required to transform a partial base layer circuit into a full one (merging structures) for WVG and proving.
These changes can significantly reduce peak RAM usage in BWG, while speeding up the slowest part of the witness generation process.
Checklist
zk fmtandzk lint.