The effect of opening a large number of (sub) databases on performance

[azarenkovgd]

Hi!

How correct is it to open thousands (or even tens of thousands and more) of (sub) databases?
As I understand it, technically it is possible (by increasing the limit in the configuration), but I'm not sure that it is correct from the performance point of view.

1. Does the number of open databases affect the performance of each database individually (read and write)?
2. Is there any physical limit to the number of open (sub) databases?
3. Whether the alternative is to instead open a large number of (root) databases (each in its own folder etc.) ?

Thank you!

[kriszyp]

The maximum number of databases does affect performance, according to the LMDB docs:

Currently a moderate number of slots are cheap but a huge number gets expensive: 7-120 words per transaction, and every mdb_dbi_open() does a linear search of the opened slots.

http://www.lmdb.tech/doc/group__mdb.html#gaa2fc2f1f37cb1115e733b62cab2fcdbc
(this function is mapped maxDbs)
However, I think it requires a very large number before there is any noticeable impact (probably tens of thousands). I think this is stored in a 32-bit value, so might be a 4G limit on number, but almost certainly would run into performance issues before then.
Generally each root database is going to be much more expensive than a sub database, but if you have thousands of databases it is probably better to subdivide them into several root databases if possible.

[azarenkovgd]

Thank you!