Add ReadCache option to LevelDB and RocksDB

[Qiao-Jin]

This PR adds a "ReadCache" option to LevelDB and RocksDb, which determines whether db will employ read cache (currently no).

This option helps to cache db data across block priod. Currently snapshot only supports data cache within a block period.

[shargon]

There are any cons?

[Qiao-Jin]

There are any cons?

There should be benefits, like if there exist some often-called address / contracts. Currently snapshot only serves within one block period, or in other words, need to fetch same piece of data from db again even if it has been fetched in last block. This PR adds readcache to system and will solve such problems.

[erikzhang]

This has a big disadvantage. Its cache will be duplicated with DataCache, thereby reducing performance.

[Qiao-Jin]

This has a big disadvantage. Its cache will be duplicated with DataCache, thereby reducing performance.

The default LRU cache size of levelDB is only 8 MB, which will not influence performance I think. Furthermore after persisting block snapshot will be flushed and we will need to fetch any data from harddisk again, if readcache is not enabled.

[erikzhang]

Has it been tested? Has performance improved?

[Qiao-Jin]

Has it been tested? Has performance improved?

TPS is almost the same, but this is due to monotonous transaction data. Its effect should be better in real world env where the size of cached data is much bigger.

[vncoelho]

@joeqian10, perhaps if we are not sure about the performance we can set ReadCache=false as default and merge the PR, when we have more experiments we change the default value.

[joeqian10]

@vncoelho I think you are @Qiao-Jin here, hahaha😂

[Qiao-Jin]

@joeqian10, perhaps if we are not sure about the performance we can set ReadCache=false as default and merge the PR, when we have more experiments we change the default value.

Yes I think so.

[erikzhang]

perhaps if we are not sure about the performance we can set ReadCache=false as default and merge the PR, when we have more experiments we change the default value.

Or should we test it first before merge?

[vncoelho]

I think we should test if it works, @erikzhang, however, we need a better setup for general tests on performance. We have been working on this and NGD did good experiments in that PRs about the ConsensusPrepareRequest caching neo-project/neo#949.

I have some scripts in R that automatically generates nice graphs and test like ANOVA for comparisons in multiple axis, I used all them in my thesis and other works about optimization. We need to create some kind of repository that creates this automatic performance analysis in the PRs, in which we just pass commits as references and it compares intensively. That would surely brings us a lot of tranquility in decision making.