feat: Qdrant vector search support

[Anush008]

Description:

This PR adds support for Qdrant to be used for vector search.

Checklist

• Do new or existing unit or integration tests cover this code?
• Did you run make unit_testing and make integration-testing successfully?
• Do new classes, functions, methods and parameters all have docstrings?
• Were existing docstrings updated, if necessary?

[blythed]

Hi @Anush008 - thank-you for your work and this important contribution to the project!

My main concerns are around configuring the solution (in the config.py module) and
also how we test this.

Could you add comments in the PR on how you tested this? Also, we should somehow test this in the CI too.

[Anush008]

Hey @blythed. The implementation already undergoes the unit tests..
By default, it uses an :in-memory: instance of the Qdrant API.

[Anush008]

In-memory mode reference.

[jieguangzhou]

Great work!

[blythed]

@Anush008 could you get the CI passing?

[Anush008]

Haha. I assumed it was some flaky test. Updated the CHANGELOG. Should be good now.

[Anush008]

@blythed, could you please approve the CI to check?

[Anush008]

I am not sure if the failure is related to this PR.

[fnikolai]

Thanks @Anush008 for your PR.

This error usually occurs due to implicit type casting, such as using a string where an int is expected, or vice versa.

It would be very helpful if you could identify where this type casting happens (e.g., in the arguments used when calling qdrant).

If pinpointing the issue is not feasible, as a last resort, you can try increasing the type_ignore value in test/unittest/test_quality.py.

Here's the link to the relevant line:

superduper/test/unittest/test_quality.py

Line 22 in 875ac4f

'type_ignore': 10, # This should only ever increase in obscure edge cases

[jieguangzhou]

#2428 (comment)

Thanks for your PR @Anush008 .

Please fix the CI, then we can merge it

[Anush008]

Pushed a fix. Should be green now.

[Anush008]

cc @fnikolai @jieguangzhou