+mdb atlas vectordb [clean_final]

[ranfysvalle02]

Why are these changes needed?

MongoDB has been ranked as the best vector database(https://www.mongodb.com/blog/post/atlas-vector-search-commands-highest-developer-nps-retool-state-ai-2023-survey) in the Retool AI report, so it is quite important to add MongoDB vector search as an option for Autogen RAG.

You can easily start the MongoDB vector search on a free tier M0 MongoDB Atlas cluster. Free tier cluster provides the full functionality of the MongoDB vector search. https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/

But why is MongoDB such a standout? Well, there are a few key reasons.

MongoDB Atlas integrates smoothly with existing databases. For organizations already using MongoDB, this means a seamless expansion into the vector storage—no major system overhauls required!
MongoDB Atlas is built to handle operational heavy-lifting. It excels when serving large-scale, mission-critical applications, offering robustness and reliability where it counts.
MongoDB's flexibility in handling a variety of data types and structures makes it perfectly suited to the complexity of vector embeddings.

As such, implementing MongoDB as a Retrieval Agent can unlock new potential in your AI applications, bringing the full power of vector storage to bear.

Related issue number: 711

Closes #711
Closes #2996

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[ranfysvalle02]

Sorry @thinkall @Hk669 -- I know I have been a bit annoying, but I think this time we got it 👍

[thinkall]

Hi @ranfysvalle02 , thank you. I can see the issues of notebook have not been addressed. See comments here: #2996

[ranfysvalle02]

The simple change of

    assert collection.name == collection_name
    assert collection.collection_name == collection_name

Highlighted that I need to create a "wrapper" class around the MongoDB collection, similar to what pgvector did.

class Collection:
    """
    A Collection object for PGVector.

    Attributes:
        client: The PGVector client.
        collection_name (str): The name of the collection. Default is "documents".
        embedding_function (Callable): The embedding function used to generate the vector representation.
            Default is None. SentenceTransformer("all-MiniLM-L6-v2").encode will be used when None.
            Models can be chosen from:
            https://huggingface.co/models?library=sentence-transformers
        metadata (Optional[dict]): The metadata of the collection.
        get_or_create (Optional): The flag indicating whether to get or create the collection.
    """

but for MongoDB. Will be working on this @Hk669

[ranfysvalle02]

@thinkall - Can you please tell me how to 'fix' the notebook? Or perhaps have it as a 'suggested commit'? I'll be addressing the notebook and any final touches later today.

[thinkall]

@thinkall - Can you please tell me how to 'fix' the notebook? Or perhaps have it as a 'suggested commit'? I'll be addressing the notebook and any final touches later today.

What about run it successfully in your local env and remove only the sensitive info? A new user should be able to run it by fill in the missed message, which should only be the config_list.

So, the connect string of mongodb should not be empty, the one I suggested in your last PR worked for me. Does it work for you? The one you previously used didn't work for me and was not connecting to the docker container.

The output of the chat in the last cell is not correct. Could you please check my previous comments and the pgvector notebook example?

[thinkall]

The simple change of

    assert collection.name == collection_name
    assert collection.collection_name == collection_name

Highlighted that I need to create a "wrapper" class around the MongoDB collection, similar to what pgvector did.

class Collection:
    """
    A Collection object for PGVector.

    Attributes:
        client: The PGVector client.
        collection_name (str): The name of the collection. Default is "documents".
        embedding_function (Callable): The embedding function used to generate the vector representation.
            Default is None. SentenceTransformer("all-MiniLM-L6-v2").encode will be used when None.
            Models can be chosen from:
            https://huggingface.co/models?library=sentence-transformers
        metadata (Optional[dict]): The metadata of the collection.
        get_or_create (Optional): The flag indicating whether to get or create the collection.
    """

but for MongoDB. Will be working on this @Hk669

It's OK, no need to wrap a Collection class for mongodb. PGVector doesn't have a collection class, that's why we have it in AutoGen vector implementation. The key for AutoGen vectordb is to follow the protocols in base.py.

[ranfysvalle02]

I see what you mean @thinkall !

I found the issue with the notebook and notebook output!

 # MongoDB Atlas database
        },
        "get_or_create": False,  # set to False if you don't want to reuse an existing collection
        "overwrite": True,  # set to True if you want to overwrite an existing collection
    },

vs

 # MongoDB Atlas database
        },
        "get_or_create": True, 
        "overwrite": True,  # set to True if you want to overwrite an existing collection
    },

We are close!!!

I'll push the fix/code later today

[thinkall]

I see what you mean @thinkall !

I found the issue with the notebook and notebook output!

 # MongoDB Atlas database
        },
        "get_or_create": False,  # set to False if you don't want to reuse an existing collection
        "overwrite": True,  # set to True if you want to overwrite an existing collection
    },

vs

 # MongoDB Atlas database
        },
        "get_or_create": True, 
        "overwrite": True,  # set to True if you want to overwrite an existing collection
    },

We are close!!!

I'll push the fix/code later today

No errors here. I've fixed this and made a commit.

[ranfysvalle02]

@thinkall finally tracked this down --- its all about the index! the create_collection method

def create_collection(collection_name: str,
                        overwrite: bool = False,
                        get_or_create: bool = True) -> Any

does not use 'index_name' or 'similarity' -- which I had added. Working on a fix!

[ranfysvalle02]

@thinkall - I finally got it to run, but I have to add a strange programmatic arbitrary delay for things to work. I am working on a more elegant solution. After ~15seconds it works. Anything 5 seconds or less fails.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 205 lines in your changes missing coverage. Please review.

Project coverage is 11.73%. Comparing base (6279247) to head (2f1bb68).
Report is 43 commits behind head on main.

Files	Patch %	Lines
autogen/agentchat/contrib/vectordb/mongodb.py	0.00%	201 Missing ⚠️
autogen/agentchat/contrib/vectordb/base.py	0.00%	4 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (6279247) and HEAD (2f1bb68). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (6279247)	HEAD (2f1bb68)
unittests	2	1

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #3000       +/-   ##
===========================================
- Coverage   32.90%   11.73%   -21.18%     
===========================================
  Files          94       97        +3     
  Lines       10235    10807      +572     
  Branches     2193     2312      +119     
===========================================
- Hits         3368     1268     -2100     
- Misses       6580     9518     +2938     
+ Partials      287       21      -266     

Flag	Coverage Δ
unittests	11.73% <0.00%> (-21.18%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Hk669]

@thinkall - I finally got it to run, but I have to add a strange programmatic arbitrary delay for things to work. I am working on a more elegant solution. After ~15seconds it works. Anything 5 seconds or less fails.

https://github.com/microsoft/autogen/actions/runs/9629801423/job/26560601555?pr=3000 -> can you look into the tests that are failing. thanks @ranfysvalle02

[thinkall]

@Jibola , thank you very much for the updates! Could you please help review and rerun the notebook example with a high tier Atlas instance? Thanks.

[Jibola]

@Jibola , thank you very much for the updates! Could you please help review and rerun the notebook example with a high tier Atlas instance? Thanks.

Yep. Just went ahead and did so. Let me know if the new notebook suffices!

[thinkall]

Thank you so much, @Jibola , @ranfysvalle02 , @cozypet and all the other reviewers and contributors! What a great collaboration!