Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

High disk space utilization #24

Closed
akrylysov opened this issue Nov 4, 2019 · 20 comments
Closed

High disk space utilization #24

akrylysov opened this issue Nov 4, 2019 · 20 comments

Comments

@akrylysov
Copy link
Owner

Details ethereum/go-ethereum#20029.

When storing small keys/values Pogreb wastes too much space by making all writes 512-byte aligned.
@akrylysov akrylysov mentioned this issue Nov 4, 2019
Closed
@AusIV
Copy link

AusIV commented Nov 9, 2019

Any idea on a timeline for this? If you can point me in the right direction I'd be open to helping out.

@akrylysov
Copy link
Owner Author

I have some code ready, and I'll push a branch for testing in about a week. I have to change the way the data is organized on disk to not waste space and provide durability at the same time. Instead of having a single data file updated in-place, Pogreb will have multiple append-only data files (that periodically could be compacted). The indexing part will remain mostly the same, but data file management will change.

@akrylysov akrylysov mentioned this issue Nov 17, 2019
Closed
@akrylysov akrylysov added this to the 0.9 milestone Dec 7, 2019
@akrylysov
Copy link
Owner Author

@AusIV I pushed changes to the datalog branch (https://github.com/akrylysov/pogreb/tree/datalog), feel free to test it. I'll need to run more benchmarks and do more testing before merging it.

@samuelmattjohnston
Copy link

@akrylysov Hey sorry for delay on getting back to you here. We've been occupied, but I would like to check this out when I have some downtime, if you've not had a chance test it already. Have you had a chance to do any benchmarks on this yet?

@akrylysov
Copy link
Owner Author

@samuelmattjohnston the benchmarks look very promising, the space utilization issue was fixed.
I'm working on updating the design document before I can release a new version.

@samuelmattjohnston
Copy link

samuelmattjohnston commented Jan 24, 2020 •
edited
Loading

Hm. Is there some parameter I should be passing to the database for lower disk utilization? A geth database created with the same code from the github issue you linked seems to be using 7.5GB where a normal sync with leveldb seems to be using 2.5GB. -- this was on goerli.

I am running another test right now to see what it looks like on mainnet ethereum, and will post back. Hopefully it isn't more than 1TB (a normal sync is 250GB).

I'm using the datalog branch as today, in case there is another branch with more fixes I should be testing with.

@akrylysov
Copy link
Owner Author

If you have deletes or updates you should run the Compact method after loading the data, the background compaction is not enabled by default on this branch (take a look at the fields of the Options) struct.

Could you please tell more about your workload? Ratio of puts/updates/deletes, the average size of keys and values?

@samuelmattjohnston
Copy link

I'm not a database guy, unfortunately. I am a DevOps guy, but I can work with you to get that information if you've got time to help me through the code. I can send an email to you to work out something if you have time.

@AusIV
Copy link

AusIV commented Jan 24, 2020

@akrylysov - I can answer those questions for @samuelmattjohnston.

The vast majority of our operations are puts. We have about 2 updates for every 200 puts, and those are both for a key / value pair where both the key and the value are 32 bytes. Our keys are almost all 32 bytes (a small handful are smaller), and the values average 150 bytes, though they can range from 32 bytes to over 1 MB (the theoretical maximum is about 2.2 MB, though we haven't seen many over 1 MB, and the 99th percentile is around 30KB). We pretty much don't delete anything ever.

For a bit of context, we're managing Ethereum nodes, which track blockchain data. Since a blockchain is an append-only structure, we don't delete old data. The few updates we have are for a small handful of pointers that track things like the latest block and latest header.

@akrylysov
Copy link
Owner Author

@AusIV thanks for the detailed answer! In my tests the datalog branch has lower overhead per key/value than LevelDB. I'll be able to double check this weekend. I'll try storing specifically 32 byte keys and 150 byte values.

Since a blockchain is an append-only structure, we don't delete old data. The few updates we have are for a small handful of pointers that track things like the latest block and latest header.

Pogreb should work perfectly for append-only workloads. I recommend having two databases - one for the blockchain transactions that never change and another for tracking metadata like the pointers you describe. This way you will never need to run compaction on the first database. When opening the second database you should pass pogreb.Options{BackgroundCompactionInterval: time.Minute} that will periodically discard outdated values and rewrite the write-ahead log.

I'm positive we can make Pogreb work for your use cases, happy to help!

@samuelmattjohnston
Copy link

samuelmattjohnston commented Jan 24, 2020 •
edited
Loading

I have geth running pogreb in my current fork/branch: https://github.com/samuelmattjohnston/go-ethereum/tree/pogreb -- Completely in testing, and not tested much, but it is able to get up to speed with goerli. The server on mainnet seems to be taking it's time to get up to speed.

If you want to run yourself:

make geth-linux-amd64
ln -s build/bin/geth-linux-amd64 geth 
mkdir goerli
./geth --goerli --datadir ./goerli --datadirdb=pogreb # pogreb is about 7.5GB

to run mainnet:

mkdir mainnet
./geth --datadir ./mainnet --datadirdb=pogreb #TBD size -- leveldb is 300GB

Assuming linux.

mainnet is going to have larger data for each block than goerli. Goerli gets much less traffic than mainnet, but is easier and quicker to test on.

Database code lives in: ethdb/pogreb/pogreb.go. Haven't cleaned up comments from the code that I copied from the upstream implementation of pogreb.

@akrylysov
Copy link
Owner Author

Thanks. I tested storing 100k items using https://github.com/akrylysov/pogreb-bench. The results are:

$ ./pogreb-bench -d test -n 100000 -mink 32 -maxk 32 -minv 150 -maxv 150 -e pogreb
File size: 20.86MB

$ ./pogreb-bench -d test -n 100000 -mink 32 -maxk 32 -minv 150 -maxv 150 -e goleveldb
File size: 20.71MB

$ ./pogreb-bench -d test -n 100000 -mink 32 -maxk 32 -minv 150 -maxv 150 -e badger
File size: 26.21MB

$ ./pogreb-bench -d test -n 100000 -mink 32 -maxk 32 -minv 150 -maxv 150 -e pogreb
File size: 40.01MB

go.mod is pointing to pogreb v0.8.3 (https://github.com/samuelmattjohnston/go-ethereum/blob/pogreb/go.mod#L12) could that be the issue?

@samuelmattjohnston
Copy link

I had that already changed to the datalog branch but some reason that got reset.. Thanks for the catch. I'll redo my testing now.

@samuelmattjohnston
Copy link

samuelmattjohnston commented Jan 27, 2020 •
edited
Loading

just a quick update, 5GB usage now for pogreb on goerli. I've just pushed up the go.mod fix.
I'll have to restart the server I've had trying to sync for the past couple of days and will post when that is done with a full sync of mainnet.

@samuelmattjohnston
Copy link

Mainnet is giving me issues. I seem to be getting some errors when there is a rollback of headers..

INFO [01-27|02:48:03.689] Imported new state entries               count=1590 elapsed=16.932ms  processed=15714925 pending=88184  retry=0   duplicate=2748 unexpected=35501
WARN [01-27|02:48:03.700] Rolled back headers                      count=2048 header=2076175->2076175 fast=2036697->2038745 block=0->0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x9741e3]
goroutine 162278 [running]:
github.com/ethereum/go-ethereum/core.(*HeaderChain).WriteHeader(0xc0001742d0, 0xc06ce84900, 0x639836e40b982218, 0xcc2e4330ef4d36fd, 0xacef7041905f1182)
        /ext-go/1/src/github.com/ethereum/go-ethereum/core/headerchain.go:179 +0x833
github.com/ethereum/go-ethereum/core.(*BlockChain).InsertHeaderChain.func1(0xc06ce84900, 0x99a5f2c6b60775ef, 0x639836e40b982218)
        /ext-go/1/src/github.com/ethereum/go-ethereum/core/blockchain.go:2112 +0x38
github.com/ethereum/go-ethereum/core.(*HeaderChain).InsertHeaderChain(0xc0001742d0, 0xc056a99000, 0x300, 0x300, 0xc0006d5b90, 0xbf83b1b61b30cdef, 0x15cfd03b7b1, 0x2158740, 0xc000178140, 0xd8200aa15cd33ad4, ...)
        /ext-go/1/src/github.com/ethereum/go-ethereum/core/headerchain.go:287 +0x2c9
github.com/ethereum/go-ethereum/core.(*BlockChain).InsertHeaderChain(0xc0008c8000, 0xc056a99000, 0x300, 0x300, 0x64, 0x0, 0x0, 0x0)
        /ext-go/1/src/github.com/ethereum/go-ethereum/core/blockchain.go:2115 +0x224
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).processHeaders(0xc00029c000, 0x1fac10, 0x8ed7b0, 0xc077978460, 0x0, 0x0)
        /ext-go/1/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1469 +0x6a3
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer.func6(0xc062997f70, 0x3)
        /ext-go/1/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:517 +0x48
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1(0xc00029c000, 0xc066ffade0, 0xc062e05890)
        /ext-go/1/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:534 +0x63
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync
        /ext-go/1/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:534 +0xaf

if I try to resume:

$ sudo -u geth /usr/bin/geth  --datadir=/var/lib/ethereum --datadirdb=pogreb --maxpeers "100" 
WARN [01-27|16:21:16.276] Sanitizing cache to Go's GC limits       provided=4096 updated=2575
INFO [01-27|16:21:16.277] Maximum peer count                       ETH=100 LES=0 total=100
INFO [01-27|16:21:16.277] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[01-27|16:21:16.277] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[01-27|16:21:16.278] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[01-27|16:21:16.278] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[01-27|16:21:16.278] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[01-27|16:21:16.278] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[01-27|16:21:16.278] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
INFO [01-27|16:21:16.279] Starting peer-to-peer node               instance=Geth/v1.9.9-stable-8439aaf7/linux-amd64/go1.13.4
INFO [01-27|16:21:16.279] Allocated trie memory caches             clean=643.00MiB dirty=643.00MiB
INFO [01-27|16:21:16.279] Starting with pogrebdb Pog 
pogreb: moving non-data files...
Fatal: Error starting protocol stack: rename /var/lib/ethereum/geth/chaindata/pogreb.db/main.index /tmp/pogreb_recovery943887997/main.index: invalid cross-device link

I'm not sure what the cause is atm but if I get time I'll be working on it again.

@akrylysov
Copy link
Owner Author

Thanks, I'll test your branch of go-ethereum this weekend. Here is an updated design document if you are curious https://github.com/akrylysov/pogreb/blob/datalog/docs/design.md.

@AusIV
Copy link

AusIV commented Jan 31, 2020

For what it's worth, /var/lib/ethereum is one mount, /tmp/ is another. If there's a way we can tell pogreb to use the /var/lib/ethereum mount for recovery I think that's what we need.

@akrylysov
Copy link
Owner Author

Thanks for the bug report, I updated the branch to not use /tmp on recovery.

@akrylysov
Copy link
Owner Author

I'm having difficulties testing go-ethereum. geth panics after some time. Pogreb is not in the stack trace, so I'm not sure if it's related:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x9741e3]

goroutine 924 [running]:
github.com/ethereum/go-ethereum/core.(*HeaderChain).WriteHeader(0xc0000d6360, 0xc00c793680, 0x24f5eeb807ff9a02, 0xc23968b443ad78bb, 0xb7a5bd28be6979b1)
        /ext-go/1/src/github.com/ethereum/go-ethereum/core/headerchain.go:179 +0x833
github.com/ethereum/go-ethereum/core.(*BlockChain).InsertHeaderChain.func1(0xc00c793680, 0xce574620d0775aca, 0x24f5eeb807ff9a02)
        /ext-go/1/src/github.com/ethereum/go-ethereum/core/blockchain.go:2112 +0x38
github.com/ethereum/go-ethereum/core.(*HeaderChain).InsertHeaderChain(0xc0000d6360, 0xc000174000, 0x240, 0x240, 0xc000789b90, 0xbf88f3db1ea8c224, 0x751e9b311f, 0x2159740, 0xc0005be120, 0x840b7a5f2c629239, ...)
        /ext-go/1/src/github.com/ethereum/go-ethereum/core/headerchain.go:287 +0x2c9
github.com/ethereum/go-ethereum/core.(*BlockChain).InsertHeaderChain(0xc00076c000, 0xc000174000, 0x240, 0x240, 0x64, 0x0, 0x0, 0x0)
        /ext-go/1/src/github.com/ethereum/go-ethereum/core/blockchain.go:2115 +0x224
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).processHeaders(0xc0003d8000, 0x1, 0x21000d, 0xc00bffedc0, 0x0, 0x0)
        /ext-go/1/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1469 +0x6a3
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer.func6(0xc000663770, 0xdedfee1f92ed5114)
        /ext-go/1/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:517 +0x48
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1(0xc0003d8000, 0xc00c11c6c0, 0xc00c12c000)
        /ext-go/1/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:534 +0x63
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync
        /ext-go/1/src/github.com/ethereum/go-ethereum/eth/downloader/downloader.go:534 +0xaf

@akrylysov akrylysov mentioned this issue Mar 8, 2020
Merged
akrylysov added a commit that referenced this issue Mar 8, 2020
@akrylysov
Write-ahead log (#29)
eeec217 
Replaces the unstructured data file for storing key-value pairs with a write-ahead log.

- In the event of a crash or a power loss the database is automatically recovered (#27).
- Fixes disk space overhead when storing small keys and values (#24).
- Optional background compaction allows reclaiming disk space occupied by overwritten or deleted keys (#28).

See docs/design.md for more details.
@akrylysov akrylysov removed this from the 0.9.0 milestone Apr 3, 2020
@akrylysov
Copy link
Owner Author

Fixed in version 0.9.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@akrylysov @AusIV @samuelmattjohnston