feat: Milvus storage connector

[zc277584121]

Description
This PR adds support for Milvus - https://milvus.io/docs to be used as an archival storage connector.

Tests
Have tested successfully in local environment for tests/test_storage.py. and tests/test_load_archival.py

[zc277584121]

@sarahwooders Can you help to review this PR, thanks

[sarahwooders]

Could you please make sure you ran poetry lock with Python 3.12 and poetry version 1.8.2?

[zc277584121]

Could you please make sure you ran poetry lock with Python 3.12 and poetry version 1.8.2?

@sarahwooders Thank you for reminding me, and I think it's the correct setting now.

[sarahwooders]

Could you please make sure you ran poetry lock with Python 3.12 and poetry version 1.8.2?

@sarahwooders Thank you for reminding me, and I think it's the correct setting now.

Could you please double check you are using the right versions? It looks like the tests are still failing.

[zc277584121]

Could you please make sure you ran poetry lock with Python 3.12 and poetry version 1.8.2?

@sarahwooders Thank you for reminding me, and I think it's the correct setting now.

Could you please double check you are using the right versions? It looks like the tests are still failing.

Hi, @sarahwooders
I'm sure that I'm using the right version.

I guess there may be other problems that caused this test to fail.

The tests failed due to hash poetry.lock file failing, I don't know what it means, I added milvus dependency in pyproject.toml, and updated poetry.lock file. Does this hash error indicate that I cannot modify the lock file? Or indicate other problems?

Can you please help to confirm or provide more information, thanks a lot.

[sarahwooders]

Could you please give me edit access to your branch, and i can play around with it? Not sure what is causing the error.

[zc277584121]

@sarahwooders I have invited you as collaborator into my repo. Thanks for helping to check the problem

[zc277584121]

Any progress on the debugging? @sarahwooders

[zc277584121]

I added another commit "adapt to milvus lite" where deleted the milvus service starting from docker in githubworkflow file, instead, I use milvus lite to start milvus without any other new shell script. In this way, the hash file problem is fixed. However, there are still some problem with milvus lite, we will try to fix them as soon as possible.

[sarahwooders]

lgtm - merging into a staging branch so we can work on fixing some of the tests.

[zc277584121]

@sarahwooders Thanks for the merging into milvus-staging. And here is my new pr: #1405. I updated the pymilvus and milvus-lite dependency, and made pymilvus optional. Now in the case of memgpt, the latest version of milvus-lite get the same capabilities as milvus docker service does. And I think it's currently good enough to be merged into main branch. Could you please help to review it?