Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(en): Improve tree snapshot recovery #1938

Merged
merged 9 commits into from
May 29, 2024
Merged

feat(en): Improve tree snapshot recovery #1938

merged 9 commits into from
May 29, 2024

Conversation

slowli
Copy link
Contributor

@slowli slowli commented May 15, 2024 •
edited
Loading

What ❔

  • Adds more logs that will allow to track tree recovery progress more clearly.
  • Uses tagged DB connections.
  • Makes chunk size configurable.

Why ❔

  • Logs and tagged connections improve observability.
  • Configuring chunk size during tree recovery allows to fine-tune its performance (theoretically; needs to be tested on E2E tests).

Checklist

  • PR title corresponds to the body of PR (we generate changelog entries from PRs).
  • Tests for the changes have been added / updated.
  • Documentation comments have been added / updated.
  • Code has been formatted via zk fmt and zk lint.
  • Spellcheck has been run via zk spellcheck.

@slowli slowli marked this pull request as ready for review May 15, 2024 12:46
@tomg10
Copy link
Contributor

tomg10 commented May 15, 2024

I'm not sure about this PR, I don't intuitively see why this could impact the performance of the tree recovery much and us fine-tuning this config value will take time we should spend working on tree-less mode.

@slowli
Copy link
Contributor Author

slowli commented May 15, 2024

us fine-tuning this config value will take time we should spend working on tree-less mode.

Come on, it's changing a value once per day in the E2E snapshot recovery workflow config and looking at results (this second part can be deferred). I don't think it would require much time if we set up it this way.

@tomg10
Copy link
Contributor

tomg10 commented May 15, 2024

It's still much lower priority work, this sounds like P3 at best, while tree-less mode is P0/P1. The situation would look totally different if treeless progress was blocked by some external factors and this task was done as a "time filler", but as I understand this is not the case right now.

@slowli slowli added this pull request to the merge queue May 29, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 29, 2024
@slowli slowli added this pull request to the merge queue May 29, 2024
Merged via the queue into main with commit 5bc8234 May 29, 2024
44 checks passed
@slowli slowli deleted the aov-pla-933-improve-tree-snapshot-recovery branch May 29, 2024 08:25
@zksync-era-bot zksync-era-bot mentioned this pull request May 29, 2024
Merged
github-merge-queue bot pushed a commit that referenced this pull request May 30, 2024
@zksync-era-bot
chore(main): release core 24.5.0 (#2004)
e0923ff 
🤖 I have created a release *beep* *boop*
---


##
[24.5.0](core-v24.4.0...core-v24.5.0)
(2024-05-30)


### Features

* Add protocol_version label to WG jobs metric
([#2009](#2009))
([e0a3393](e0a3393))
* **config:** remove zksync home
([#2022](#2022))
([d08fe81](d08fe81))
* **en:** Improve tree snapshot recovery
([#1938](#1938))
([5bc8234](5bc8234))
* Make house keeper emit correct protocol version
([#2062](#2062))
([a58a7e8](a58a7e8))
* **node_framework:** Migrate main node to the framework
([#1997](#1997))
([27a26cb](27a26cb))
* **node_framework:** Synchronize pools layer with logic in
initialize_components
([#2079](#2079))
([3202461](3202461))
* Protocol semantic version
([#2059](#2059))
([3984dcf](3984dcf))
* **prover:** Adnotate prover queue metrics with protocol version
([#1893](#1893))
([d1e1004](d1e1004))
* save writes needed for tree in state keeper
([#1965](#1965))
([471af53](471af53))
* **test:** Add filebased config support for integration tests
([#2043](#2043))
([be3ded9](be3ded9))
* **vm-runner:** implement VM runner main body
([#1955](#1955))
([bf5b6c2](bf5b6c2))


### Bug Fixes

* **API:** polish web3 api block-related types
([#1994](#1994))
([6cd3c53](6cd3c53))
* **en:** chunk factory deps
([#2077](#2077))
([4b9e6fa](4b9e6fa))
* **en:** Fix recovery-related metrics
([#2014](#2014))
([86355d6](86355d6))
* **eth-watch:** Do not track for stm, only for diamond proxy
([#2080](#2080))
([87adac9](87adac9))
* fix metrics reporting wrong values
([#2065](#2065))
([2ec010a](2ec010a))
* **loadtest:** resolve unit conversion error in loadtest metrics
([#1987](#1987))
([b5870a0](b5870a0))
* **merkle-tree:** Fix incoherent Merkle tree view
([#2071](#2071))
([2fc9a6c](2fc9a6c))
* **metadata-calculator:** protective reads sort
([#2087](#2087))
([160c13c](160c13c))
* **node_framework:** Fix the connection pool size for the catchup task
([#2046](#2046))
([c00a2eb](c00a2eb))
* **node_framework:** Use custom pool for commitiment generator
([#2076](#2076))
([994df8f](994df8f))
* **protocol_version:** Add backward compatibility
([#2097](#2097))
([391624b](391624b))
* **pruning:** Fix DB pruner responsiveness during shutdown
([#2058](#2058))
([0a07312](0a07312))
* **zk_toolbox:** Use both folders for loading contracts
([#2030](#2030))
([97c6d5c](97c6d5c))


### Performance Improvements

* **commitment-generator:** Run commitment generation for multiple
batches in parallel
([#1984](#1984))
([602bf67](602bf67))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: zksync-era-bot <zksync-era-bot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@popzxc popzxc popzxc approved these changes

@RomanBrodetski RomanBrodetski Awaiting requested review from RomanBrodetski

@tomg10 tomg10 Awaiting requested review from tomg10

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@slowli @tomg10 @popzxc