Switching to RocksDB as the default db backend

[kwunyeung]

Closes #97

The latest version of Tendermint added an option for using RocksDB as the db backend.

tendermint/tendermint#4239

which is available in SDK v0.38.x

RocksDB was created by Facebook as a high performance implementation of LevelDB.

https://github.com/facebook/rocksdb/wiki

Some benchmarks in earlier years have shown that although RocksDB would create a large DB in size but with better performance in read/white/delete operations.

https://www.influxdata.com/blog/benchmarking-leveldb-vs-rocksdb-vs-hyperleveldb-vs-lmdb-performance-for-influxdb/

With a similar use case as Facebook, I suggest we should try using RocksDB as the default db backend.

Of course we have to study if there is any drawback after adapting it. How the performance gain cost resources usage, would it increase difficulties running fullnodes, etc.

[RiccardoM]

Finding and considerations about this improvement

After taking a look at the implementation that has been done inside #154, I would like to recap what I've discovered about the different database backends that are used inside a Cosmos-SDK application.

1. Different databases

The main thing to understand is that there are different databases being used.

Specifically, there are more than one used by Tendermint and only one (as far as I can tell) used by Cosmos.

2. Different database engines

Since the Tendermint and Cosmos databases are created in different points in time, they can run on different database engines.

3. Different ways of specifying the database engines

Since they can be different between Tendermint and Cosmos, there are two ways of specifying the database engine to be used. Particularly:

1. For Tendermint databases, this is done inside the ~/.desmosd/config/config.toml file using the db_backend atribute.
   E.g. db_backend="goleveldb"

2. For the Cosmos database, this is done using the -X 'github.com/cosmos/cosmos-sdk/types.DBBackend=' ldflag when building the binaries.
   E.g. make build LDFLAGS="-X 'github.com/cosmos/cosmos-sdk/types.DBBackend='goleveldb' "

4. Different database backends support

Due to the fact that they are created in different ways, the backends they support is very different:

1. For Tendermint databases, they support goleveldb, cleveldb, memdb, boltdb and rocksdb.

2. For Cosmos databases, they currently support only goleveldb and cleveldb.

To solve this difference I've raised issue #6218 and created PR #6219 on Cosmos to support the same backends that Tendermint supports.

5. Using a database different than goleveldb

Finally, I've tried first to change only the Tendermint database backend, and then to change event the Cosmos database backend by using a modified version of the SDK (the same that I've requested to be merged into the Cosmos SDK).

The first time I tried building the Desmos binaries to be later able to use RocksDB as the Tendermint backend engine using the following command

make install BUILD_TAGS="gcc rocksdb"

I got the following error:

fatal error: rocksdb/c.h: No such file or directory

After Googling for the solution, I found out that

In order to be used, any custom database backend which is not in pure Go (this includes ClevelDB, BoltDB and RocksDB) needs to be installed on the machine that is compiling the binaries.

Once I installed RocksDB (see below) I was able to properly building the Desmos binaries by using the above mentioned command.

I also tried telling Cosmos to use RocksDB as its database backend running

make build \
  BUILD_TAGS="gcc rocksdb" \
  LDFLAGS="-X 'github.com/cosmos/cosmos-sdk/types.DBBackend=rocksdb'"

And it worked!

Installing RocksDb

In order to properly install RocksDB the following commands should be used.
(Further instructions can be found here)

Ubuntu

$ sudo apt-get install -y libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev libzstd-dev liblz4-dev
$ git clone https://github.com/facebook/rocksdb.git
$ cd rocksdb
$ DEBUG_LEVEL=0 make shared_lib
$ sudo make install-shared

MacOS

$ brew install rocksdb

Conlusions

I think that switching to RocksDB as a database backend is surely doable, although I do not recommend making it the default option as it requires the user to install it on its machine, which is not that easy of a job.

I think that making it optional might be a good chance to even have different nodes running different databases backends.

Side notes

I've pushed all my progress to the leonardo/rocksdb-impl branch. I've made so that using the make install DB_BACKEND=rocksdb command it compiles the binaries in a way that allow the user to use the RocksDB engine on both Tendermint and the Cosmos SDK.

Please note that in order to run this command you need to have RocksDB installed locally as described above.

[kwunyeung]

Thanks for the research. As long as it would work and won't give any conflicts in the Cosmos SDK, I would like to make it a suggestion when the node operators deploy Desmos. It doesn't have to be a default option. Installing RocksDB seems quite straight forward. I see more and more projects (at least Solana and Oasis, Lino before they switched to TRON) are using RocksDB for key-value pair database. I believe there are reasons behind this trend and these projects are expecting high read/write throughput. The performance of the DB also affect the sync speed. Can we do some more test and benchmarking on the performance between GolevelDB and RocksDB?

[leobragaz]

I'm implementing benchmarks tests right now, once I've finished I could try some tests on different local chains configurations. What do you think @RiccardoM @kwunyeung ?

[RiccardoM]

In my personal opinion, I think that actually there is no need to implement our custom benchmark tests for this. A part for being extremely costly, it would add pretty much no value to the currently existing tests are have already been made by other people. We could for example refer to:

• Benchmarking LevelDB vs. RocksDB vs. HyperLevelDB vs. LMDB Performance for InfluxDB
• Go K/V Databases Benchmark

Just by looking at those it appears that RocksDB is much faster than LevelDB, and that's why I would suggest validators to use, without making it the default one (due to the long time of compilation required).

[kwunyeung]

Ok good to go!