Wrong RocksDB.Options parameter

[iFA88]

Greetings!

I have discovered a possibly mistake parameter who the parity initializes the RocksDB. The latest parity what i use: Parity-Ethereum/v2.7.0-nightly-cd8e8c4-20191031/x86_64-linux-gnu/rustc1.38.0 in archive mode.
The RocksDB LOG file says the following:
Initializing default column family:

Options.max_bytes_for_level_base: 268435456
Options.level_compaction_dynamic_level_bytes: 0

Initializing other columns family (col0 - col8) :

Options.max_bytes_for_level_base: 14913080
Options.level_compaction_dynamic_level_bytes: 1

14.9 mb is way to low for storing terrabytes of data on 7 level.
The RocksDB engine gives also warnings about this:

2019/11/01-17:21:40.701005 7f2831e9f600 [WARN] [/ci-cache/parity-ethereum/cargo/build-linux/registry/src/github.com-1ecc6299db9ec823/parity-rocksdb-sys-0.5.6/rocksdb/db/version_set.cc:2485] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2019/11/01-17:21:40.701286 7f2831e9f600 [WARN] [/ci-cache/parity-ethereum/cargo/build-linux/registry/src/github.com-1ecc6299db9ec823/parity-rocksdb-sys-0.5.6/rocksdb/db/version_set.cc:2485] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
2019/11/01-17:21:40.701753 7f2831e9f600 [WARN] [/ci-cache/parity-ethereum/cargo/build-linux/registry/src/github.com-1ecc6299db9ec823/parity-rocksdb-sys-0.5.6/rocksdb/db/version_set.cc:2485] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.

I think the max_bytes_for_level_base parameter are accidentally mixed up. RocksDB recommendation for this value for big databases are 512MB ( https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide )

I'm not an expert in RocksDB, but i think because of this parameter the auto compact module compact a lot of tables unnecessary. Today have started my client in offline mode the check what RocksDB works without any syncing. They have done about thousands of:

2019/11/01-18:04:24.301933 7f281f5fb700 EVENT_LOG_v1 {"time_micros": 1572627864301924, "job": 358, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [2151661], "files_L2": [2151652, 2151653, 2151654], "score": 1.17118, "input_data_size": 155355733}

compacting. It has run 4.5 hours until they has idle.

This can significant increase db corruption and lot of IO work for nothing.

CF RocksDB statistics:

** Compaction Stats [default] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
 Sum      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
** Compaction Stats [col0] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0   16.21 MB   1.1      0.0     0.0      0.0       0.9      0.9       0.0   1.0      0.0     96.8        10        48    0.199       0      0
  L1      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.9   1.0    104.3    104.2         0         1    0.316     90K     45
  L2      2/0   98.37 MB   0.7      5.8     0.9      4.9       5.8      0.9       0.0   6.6     96.0     95.9        62        46    1.350     16M    10K
  L3     30/0    1.35 GB   1.0      8.1     0.9      7.2       8.1      0.9       0.0   9.2     78.8     78.7       105        14    7.482     20M    15K
  L4    291/0   13.88 GB   1.0      9.6     0.9      8.8       9.6      0.9       0.0  11.0     78.4     78.4       126        14    8.981     10M    14K
  L5   3101/0   138.88 GB   1.0      7.3     0.8      6.5       7.3      0.8       0.0   8.9     79.1     79.1        94        13    7.244   2263K    11K
  L6  45285/0    2.74 TB   0.0     12.0     0.8     11.2      12.0      0.8       0.0  14.8     90.6     90.5       136        13   10.468   2252K    11K
 Sum  48710/0    2.89 TB   0.0     42.9     4.3     38.6      43.7      5.2       0.9  48.5     82.4     84.1       533       149    3.575     52M    63K
 Int      0/0    0.00 KB   0.0     10.1     1.0      9.2      10.3      1.2       0.2  47.3     59.9     61.2       173        35    4.942   9798K    16K
** Compaction Stats [col1] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0   26.54 KB   0.5      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0      1.4         0        23    0.019       0      0
  L3      1/0   776.26 KB   0.1      0.0     0.0      0.0       0.0      0.0       0.0   9.7     14.6     14.6         0        11    0.035     10K      0
  L5      6/0   308.67 MB   1.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
  L6     51/0    3.08 GB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
 Sum     59/0    3.38 GB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  10.3      6.8      7.5         1        34    0.024     10K      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0  11.8     10.3     11.2         0         7    0.020    2806      0
** Compaction Stats [col2] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0   703.06 KB   0.5      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0     15.4         1        23    0.053       0      0
  L2      1/0    6.30 MB   0.4      0.1     0.0      0.1       0.1      0.0       0.0   4.9     51.5     51.4         2        11    0.157    5121      0
  L3      1/0   46.02 MB   0.7      0.0     0.0      0.0       0.0      0.0       0.0   3.0     92.8     92.8         0         1    0.496    3187      0
  L4     14/0   600.08 MB   0.9      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
  L5    124/0    6.24 GB   1.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
  L6   1017/0   62.79 GB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
 Sum   1158/0   69.66 GB   0.0      0.1     0.0      0.1       0.2      0.1       0.0   8.2     39.2     44.6         3        35    0.099    8308      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   4.1     27.5     36.5         1         9    0.057     888      0
** Compaction Stats [col3] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0   901.41 KB   0.5      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0     29.2         1        21    0.034       0      0
  L2      1/0   13.18 MB   0.9      0.1     0.0      0.1       0.1      0.0       0.0   4.7     33.6     35.0         3        10    0.265    583K     38
  L3      1/0   22.09 MB   0.3      0.1     0.0      0.1       0.1      0.0       0.0   5.9     80.1     80.1         1         1    1.076    972K      1
  L4     14/0   759.39 MB   1.0      0.3     0.1      0.3       0.3      0.1       0.0   5.0     86.4     86.3         4         1    3.706   4411K      9
  L5    145/0    7.67 GB   1.0      0.3     0.1      0.2       0.3      0.1       0.0   4.0     89.0     88.9         3         1    2.878    884K      0
  L6   1252/0   77.05 GB   0.0      0.4     0.1      0.3       0.4      0.1       0.0   6.0     79.8     79.7         5         1    4.818    884K     19
 Sum   1414/0   85.50 GB   0.0      1.1     0.2      0.9       1.1      0.2       0.0  55.8     71.7     73.2        16        35    0.452   7736K     67
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   8.7     18.4     21.6         2         8    0.219    205K     12
** File Read Latency Histogram By Level [col3] **

** Compaction Stats [col4] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0      0.1         0         2    0.013       0      0
  L6      1/0    0.98 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.5      0.1      0.0         0         1    0.030       6      4
 Sum      1/0    0.98 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.5      0.1      0.1         0         3    0.019       6      4
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
** Compaction Stats [col5] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0   179.32 KB   0.5      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0      9.0         1        23    0.025       0      0
  L6      1/0   372.51 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.8      8.2      3.7         1        11    0.099   1979K   537K
 Sum      2/0   551.83 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.8      5.4      5.5         2        34    0.049   1979K   537K
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.9      3.5      3.4         1         9    0.076    537K   144K
** Compaction Stats [col6] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0    0.94 KB   0.5      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0      0.0         0         4    0.023       0      0
  L6      1/0    0.93 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.5      0.1      0.0         0         2    0.027       6      4
 Sum      2/0    1.87 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.5      0.0      0.0         0         6    0.025       6      4
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0      0.0         0         1    0.019       0      0
** Compaction Stats [col7] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
 Sum      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
** Compaction Stats [col8] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
 Sum      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0

[grbIzl]

Thanks for the issue! From the brief analysis with @ordian it seems, that we don't setup this parameter by ourselves.

CC @ordian @andresilva @arkpar

[niklasad1]

@grbIzl @ordian

Is this fixed by paritytech/parity-common#256?

[ordian]

Fixed by #11248