Problem: cronosd high disk io consumption #291

barryz · 2022-01-05T02:33:07Z

Describe the bug
Several cronosd nodes we have run in our private network. Each of them has a vast disk IO consumption. My question is, the high disk io usage should be expected or not?

Expected behavior

Screenshots

Additional context
cpu.profile.zip

allthatjazzleo · 2022-01-14T06:19:48Z

hi @barryz
We also encountered this issue actually. it is probably due to this issue and should be fixed by this PR in next upgrade.
Currently, we also experiment the rocksdb backend for cronosd. It turns out lower I/O util when serving eth-rpc requests than default goleveldb.
If you are interested in runnig cronosd with rocksdb, you can check out this branch and build cronosd for mainnet binary.

COSMOS_BUILD_OPTIONS=rocksdb make build

As this branch also include the commit for (Problem: eth_getLogs with large block range is slow and cause OOM), you need to add following line in app.toml json-rpc directive

[json-rpc]
.
.
.
# LogsCap defines the max number of results can be returned from single 'eth_getLogs' query.
logs-cap = 10000

# BlockRangeCap defines the max block range allowed for 'eth_getLogs' query.
block-range-cap = 10000

config.toml

db_backend = "rocksdb"

We also found cronosd with rocksdb generally use more memory, so provision 32GB memory will be sufficient.

barryz · 2022-01-14T13:41:13Z

Thanks for your reply. Once the binary with rocksdb is built, should I resync data from scratch?

yihuang · 2022-01-14T13:54:27Z

Thanks for your reply. Once the binary with rocksdb is built, should I resync data from scratch?

yes

barryz · 2022-01-14T14:20:09Z

Thanks for your reply. Once the binary with rocksdb is built, should I resync data from scratch?

yes

A panic error shows while I re-syncing with built binary by following this tutorial. How do I fix it?

Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: 2:15PM INF asserting crisis invariants inv=6/11 module=x/crisis name=staking/module-accounts
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: 2:15PM INF asserting crisis invariants inv=7/11 module=x/crisis name=staking/nonnegative-power
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: 2:15PM INF asserting crisis invariants inv=8/11 module=x/crisis name=staking/positive-delegation
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: 2:15PM INF asserting crisis invariants inv=9/11 module=x/crisis name=staking/delegator-shares
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: 2:15PM INF asserting crisis invariants inv=10/11 module=x/crisis name=gov/module-account
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: 2:15PM INF asserted all invariants duration=0.477331 height=0 module=x/crisis
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: 2:15PM INF created new capability module=ibc name=ports/transfer
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: 2:15PM INF port binded module=x/ibc/port port=transfer
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: 2:15PM INF claimed capability capability=1 module=transfer name=ports/transfer
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: panic: unknown field "catalyst_block" in types.ChainConfig
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: goroutine 1 [running]:
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: github.com/cosmos/cosmos-sdk/codec.(*ProtoCodec).MustUnmarshalJSON(0x2979d00, {0xc000dbeb40, 0x0, 0x0}, {0x2922c30, 0xc000f14d80})
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]:         /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.5/codec/proto_codec.go:176 +0x45
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: github.com/tharsis/ethermint/x/evm.AppModule.InitGenesis({{}, _, {_, _}}, {{0x2948b08, 0xc000052038}, {0x2979d00, 0xc0011c5bc0}, {{0x0, 0x0}, ...}, ...}, ...)
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]:         /root/go/pkg/mod/github.com/crypto-org-chain/ethermint@v0.6.1-0.20211207040748-254df3803d62/x/evm/module.go:159 +0xd0
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: github.com/cosmos/cosmos-sdk/types/module.(*Manager).InitGenesis(_, {{0x2948b08, 0xc000052038}, {0x2979d00, 0xc0011c5bc0}, {{0x0, 0x0}, {0xc00102e1e0, 0x12}, 0x0, ...}, ...}, ...)
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]:         /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.5/types/module/module.go:307 +0x29d
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: github.com/crypto-org-chain/cronos/app.(*App).InitChainer(_, {{0x2948b08, 0xc000052038}, {0x2979d00, 0xc0011c5bc0}, {{0x0, 0x0}, {0xc00102e1e0, 0x12}, 0x0, ...}, ...}, ...)
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]:         /workspace/src/cronos-patch/app/app.go:771 +0x1d2
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).InitChain(0xc000591380, {{0x0, 0xed91a6d90, 0x0}, {0xc00102e1e0, 0x12}, 0xc00007d280, {0x3a67088, 0x0, 0x0}, ...})
Jan 14 14:15:52 production-cronos-mainnet-1 bash[240539]:         /root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.5/baseapp/abci.go:62 +0x475

yihuang · 2022-01-14T15:45:36Z

Thanks for your reply. Once the binary with rocksdb is built, should I resync data from scratch?

yes

A panic error shows while I re-syncing with built binary by following this tutorial. How do I fix it?

did you built with this branch? https://github.com/allthatjazzleo/cronos/tree/feature/add-rocksdb-build-from-v0.6.x?

barryz · 2022-01-14T16:15:32Z

Thanks for your reply. Once the binary with rocksdb is built, should I resync data from scratch?

yes

A panic error shows while I re-syncing with built binary by following this tutorial. How do I fix it?

did you built with this branch? https://github.com/allthatjazzleo/cronos/tree/feature/add-rocksdb-build-from-v0.6.x?

Build error

building network: mainnet
building with ledger support
go: downloading github.com/cosmos/cosmos-sdk v0.44.2
go: downloading github.com/crypto-org-chain/go-ethereum v1.10.3-patched
go: downloading github.com/yihuang/ethermint v0.6.1-0.20211228113745-556446db72eb
go: downloading github.com/crypto-org-chain/ibc-go v1.2.1-hooks
go: downloading github.com/rs/zerolog v1.25.0
go: downloading google.golang.org/genproto v0.0.0-20210909211513-a8c4777a87af
go: downloading github.com/roysc/gorocksdb v1.1.1
go: downloading golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
go: downloading golang.org/x/net v0.0.0-20210903162142-ad29c8ab022f
go: downloading github.com/improbable-eng/grpc-web v0.14.1
go: downloading github.com/palantir/stacktrace v0.0.0-20161112013806-78658fd2d177
go: downloading github.com/cosmos/iavl v0.17.1
go: downloading golang.org/x/sys v0.0.0-20210903071746-97244b99971b
go: downloading github.com/VictoriaMetrics/fastcache v1.5.7
go: downloading github.com/huin/goupnp v1.0.1-0.20210310174557-0ca763054c88
# github.com/tecbot/gorocksdb
/root/go/pkg/mod/github.com/roysc/gorocksdb@v1.1.1/array.go:4:11: fatal error: rocksdb/c.h: No such file or directory
    4 | // #include "rocksdb/c.h"
      |           ^~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:71: build] Error 2

allthatjazzleo · 2022-01-14T16:40:57Z

Thanks for your reply. Once the binary with rocksdb is built, should I resync data from scratch?

yes

A panic error shows while I re-syncing with built binary by following this tutorial. How do I fix it?

did you built with this branch? https://github.com/allthatjazzleo/cronos/tree/feature/add-rocksdb-build-from-v0.6.x?

Build error

building network: mainnet
building with ledger support
go: downloading github.com/cosmos/cosmos-sdk v0.44.2
go: downloading github.com/crypto-org-chain/go-ethereum v1.10.3-patched
go: downloading github.com/yihuang/ethermint v0.6.1-0.20211228113745-556446db72eb
go: downloading github.com/crypto-org-chain/ibc-go v1.2.1-hooks
go: downloading github.com/rs/zerolog v1.25.0
go: downloading google.golang.org/genproto v0.0.0-20210909211513-a8c4777a87af
go: downloading github.com/roysc/gorocksdb v1.1.1
go: downloading golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
go: downloading golang.org/x/net v0.0.0-20210903162142-ad29c8ab022f
go: downloading github.com/improbable-eng/grpc-web v0.14.1
go: downloading github.com/palantir/stacktrace v0.0.0-20161112013806-78658fd2d177
go: downloading github.com/cosmos/iavl v0.17.1
go: downloading golang.org/x/sys v0.0.0-20210903071746-97244b99971b
go: downloading github.com/VictoriaMetrics/fastcache v1.5.7
go: downloading github.com/huin/goupnp v1.0.1-0.20210310174557-0ca763054c88
# github.com/tecbot/gorocksdb
/root/go/pkg/mod/github.com/roysc/gorocksdb@v1.1.1/array.go:4:11: fatal error: rocksdb/c.h: No such file or directory
    4 | // #include "rocksdb/c.h"
      |           ^~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:71: build] Error 2

Hi @barryz
It should be missing rocksdb dependency.
Please follow this guide to install dependency based on your OS

For OSX, simply brew install rocksdb

allthatjazzleo · 2022-01-19T11:19:45Z

@barryz
For Linux - Ubuntu

Install rocksdb dependency:

sudo apt-get update
sudo apt-get -y install build-essential libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev libzstd-dev
git clone -b v6.27.3  https://github.com/facebook/rocksdb.git
cd rocksdb
make static_lib
sudo cp ./librocksdb.* /usr/lib/
sudo cp -r ./include/rocksdb/ /usr/include/

Build cronosd:

# back to cronos repo
CGO_LDFLAGS="-lrocksdb -lz -lbz2 -lsnappy -llz4 -lzstd" \
COSMOS_BUILD_OPTIONS=rocksdb \
make build

yihuang · 2022-03-09T04:06:54Z

in the short term, using rocksdb helps
in the middle term, the upcoming 0.7.0 upgrade should help that a lot with the statedb refactoring
in the long term, the SMT refactoring in cosmos-sdk should further optimize this too.

closing for now.

barryz changed the title ~~cronosd high disk io consumption~~ Problem: cronosd high disk io consumption Jan 5, 2022

thomas-nguy assigned allthatjazzleo Jan 11, 2022

yihuang closed this as completed Mar 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem: cronosd high disk io consumption #291

Problem: cronosd high disk io consumption #291

barryz commented Jan 5, 2022 •

edited

Loading

allthatjazzleo commented Jan 14, 2022 •

edited

Loading

barryz commented Jan 14, 2022

yihuang commented Jan 14, 2022

barryz commented Jan 14, 2022

yihuang commented Jan 14, 2022

barryz commented Jan 14, 2022

allthatjazzleo commented Jan 14, 2022 •

edited

Loading

allthatjazzleo commented Jan 19, 2022 •

edited

Loading

yihuang commented Mar 9, 2022 •

edited

Loading

Problem: cronosd high disk io consumption #291

Problem: cronosd high disk io consumption #291

Comments

barryz commented Jan 5, 2022 • edited Loading

allthatjazzleo commented Jan 14, 2022 • edited Loading

barryz commented Jan 14, 2022

yihuang commented Jan 14, 2022

barryz commented Jan 14, 2022

yihuang commented Jan 14, 2022

barryz commented Jan 14, 2022

allthatjazzleo commented Jan 14, 2022 • edited Loading

allthatjazzleo commented Jan 19, 2022 • edited Loading

yihuang commented Mar 9, 2022 • edited Loading

barryz commented Jan 5, 2022 •

edited

Loading

allthatjazzleo commented Jan 14, 2022 •

edited

Loading

allthatjazzleo commented Jan 14, 2022 •

edited

Loading

allthatjazzleo commented Jan 19, 2022 •

edited

Loading

yihuang commented Mar 9, 2022 •

edited

Loading