Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

maindb v4 #635

Closed
wants to merge 35 commits into from
Closed

maindb v4 #635

wants to merge 35 commits into from

Conversation

qianbin
Copy link
Member

@qianbin qianbin commented Dec 30, 2023 •
edited
Loading

This PR upgrades maindb to v4. It greatly reduces the storage space occupied and significantly reduces the synchronization time.

  • For MPT
    • The encoding and storage format of nodes are comprehensively improved, reducing the storage space used to store the state trie by 30%. Moreover, the speed of encoding and decoding nodes has been greatly improved.
    • Introducing the concept of Version to identify nodes, which simplifies the implementation code.
    • Optimize the trie interface to make it easier to maintain.
    • Remove leafbank stuff.
    • Improve generation-based node cache to reduce the pressure of GC.
    • Improve node storage key encoding.
  • Optimize block indexing and storage format.

Check out commits for more detail.

@codecov-commenter

This comment was marked as outdated.

Sign in to view
@qianbin qianbin changed the title WIP: maindb v4 maindb v4 Feb 25, 2024
@qianbin qianbin changed the base branch from maindb-v4 to master February 26, 2024 14:32
@qianbin qianbin requested a review from a team as a code owner February 26, 2024 14:32
@claytonneal claytonneal self-requested a review February 27, 2024 09:38
claytonneal
claytonneal requested changes Feb 27, 2024
View reviewed changes
Copy link
Member

@claytonneal claytonneal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please give more details of what testing is needed

@qianbin
Copy link
Member Author

qianbin commented Feb 27, 2024

Please give more details of what testing is needed

  1. try sync with and without -disable-pruner, and record the time and disk space used.
  2. test correctness of APIs

@qianbin
cmd/thor: update instance dir to v4
9d3f299
@qianbin
trie: implement varint-prefix coder
4cc913e
@qianbin
deps: add github.com/qianbin/drlp
8fc2d56
@qianbin
trie: implement appendHexToCompact & compactLen
af05e11
@qianbin
trie: temporarily remove merkle proof stuff
2461c69
@qianbin
trie: many changes
73cfd2f 
* disk usage reduced by 33% (force embedding shortnode)
* new encoding method for storing nodes
* optimize trie hashing
* versioning standalone nodes
* remove extended trie
* improve trie interface
* simplify NodeIterator, remove unused codes
@qianbin
trie: optimize full-node encoding/decoding
3a23814
@qianbin
trie: tweak shortnode encoding
11a235a
@qianbin
muxdb: move engine pkg
cdd32e4
@qianbin
trie: add Version() method for node interface
93877a5
@qianbin
muxdb: refactor due to trie updates and:
dd8ed4c 
* remove leafbank stuff
* simplify muxdb.Trie implementation
* improve root node cache using ttl eviction
* add leaf key filter
@qianbin
chain: a lot of changes
7822724 
* improve block content storage scheme
* remove steady block tracking
* remove tx & receipt cache
@qianbin
state: changes due to update of trie
27ba2e0
@qianbin
lowrlp: remove this pkg
20452bd
@qianbin
txpool: changes due to underlying pkg update
4ef96d4
@qianbin
genesis: changes due to underlying pkg update
c14adb2
@qianbin
consensus: changes due to underlying pkg update
2aedac0
@daithihearn
Copy link
Member

Hi @qianbin I was playing around with your changes and wanted to share some results about the sync performance.

image

@qianbin
Copy link
Member Author

qianbin commented Mar 25, 2024

Hi @qianbin I was playing around with your changes and wanted to share some results about the sync performance.

image

That's good. How about turning on -skip-logs?

@daithihearn
Copy link
Member

Hi @qianbin I was playing around with your changes and wanted to share some results about the sync performance.
image

That's good. How about turning on -skip-logs?

I'll re-run with this parameter and post the results

Hi @qianbin I was playing around with your changes and wanted to share some results about the sync performance.
image

That's good. How about turning on -skip-logs?

I'll re-run with skip logs and post the results again

@daithihearn
Copy link
Member

daithihearn commented Mar 28, 2024 •
edited
Loading

image image

@github-actions GitHub Actions
Copy link

This pull request has been marked as stale due to inactivity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale PR/ Issue has been marked as stale label Aug 18, 2024
@github-actions github-actions bot removed the stale PR/ Issue has been marked as stale label Aug 22, 2024
@github-actions GitHub Actions
Copy link

This pull request has been marked as stale due to inactivity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale PR/ Issue has been marked as stale label Oct 21, 2024
@github-actions github-actions bot closed this Oct 28, 2024
@libotony libotony reopened this Oct 29, 2024
@libotony libotony changed the base branch from master to feat/db October 29, 2024 08:18
@libotony libotony closed this Oct 29, 2024
@libotony libotony mentioned this pull request Oct 29, 2024
Open
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@claytonneal claytonneal claytonneal requested changes

@libotony libotony Awaiting requested review from libotony

Assignees
No one assigned
Labels
new feature resync-required stale PR/ Issue has been marked as stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@qianbin @codecov-commenter @daithihearn @claytonneal @libotony @darrenvechain