feat: add lmdb dbms indexer

[cristianmtr]

part of jina-ai/serve#2560

lmdb (disk-based key-value storage) indexer

more info https://lmdb.readthedocs.io/en/release/

Performance metrics

    # running lmdb benchmark with 10000 docs ...	running lmdb benchmark with 10000 docs takes 5 seconds (5.50s)
    # running lmdb benchmark with 20000 docs ...	running lmdb benchmark with 20000 docs takes 10 seconds (10.86s)
    # running lmdb benchmark with 100000 docs ...	running lmdb benchmark with 100000 docs takes 1 minute and 3 seconds (63.11s)

compared with FileDBMSIndexer (the former BinaryPb):

    # running filedbms benchmark with 10000 docs takes 12 seconds (12.42s)
    # running filedbms benchmark with 20000 docs takes 41 seconds (41.21s)
    # 100k docs never finished (stopped after 13 minutes :-1:  )

Includes:

• integration tests
• bumping jina version to latest master on all reqs.txt
• renaming Indexers

DBMSIndexer --> Indexer
QueryIndexer --> Searcher

[maximilianwerk]

Really nice. Anyhow, dump is required and vectors are currently not stored separately.

[JoanFM]

Is it possible to have both unit and integration tests?

[JoanFM]

typehint is not accurate, but actually it will never be None

[cristianmtr]

Updated

[cristianmtr]

Is it possible to have both unit and integration tests?

They are all under tests/ for each Indexer.

Integration tests are separated, since they need to import from across the repo (which you cannot do in the Dockerfile where we run the unit tests). Perhaps the problem here is that they are not just "unit" tests, since they are also tested with the Flow/ with dumping etc.

[JoanFM]

Is it possible to have both unit and integration tests?

They are all under tests/ for each Indexer.

Integration tests are separated, since they need to import from across the repo (which you cannot do in the Dockerfile where we run the unit tests). Perhaps the problem here is that they are not just "unit" tests, since they are also tested with the Flow/ with dumping etc.

Yes, there need to be unit tests without Flow

[cristianmtr]

Is it possible to have both unit and integration tests?

They are all under tests/ for each Indexer.
Integration tests are separated, since they need to import from across the repo (which you cannot do in the Dockerfile where we run the unit tests). Perhaps the problem here is that they are not just "unit" tests, since they are also tested with the Flow/ with dumping etc.

Yes, there need to be unit tests without Flow

There are