How to do indexing using postgres

[deepak-kumar-98]

            PGVector.from_documents(
                documents=documents,
                embedding=embedding_model,
                collection_name=persist_directory,
                connection_string=PGVECTOR_CONNECTION_STRING,
                use_jsonb=True,
            )

[dosubot[bot]]

To perform indexing using PostgreSQL with the PGVector library in Python, you can use the PGVector class from the LangChain library. However, note that this class is deprecated, and an improved version is available in langchain_postgres. The new implementation uses psycopg3 instead of psycopg2 and has some schema changes, so you may need to re-create your tables and re-index your data.

Here's a basic example of how to set up and use PGVector for indexing:

from langchain_community.vectorstores import PGVector
from langchain_community.embeddings.openai import OpenAIEmbeddings

# Define your connection string
CONNECTION_STRING = "postgresql+psycopg2://username:password@localhost:5432/your_database"

# Define your collection name
COLLECTION_NAME = "your_collection_name"

# Initialize your embeddings
embeddings = OpenAIEmbeddings()

# Create a PGVector instance from documents
vectorestore = PGVector.from_documents(
    embedding=embeddings,
    documents=docs,  # Replace with your list of documents
    collection_name=COLLECTION_NAME,
    connection_string=CONNECTION_STRING,
    use_jsonb=True,  # Use JSONB for metadata
)

# Add embeddings to the vector store
vectorestore.add_embeddings(
    texts=["your text data"],  # Replace with your text data
    embeddings=[[0.1, 0.2, 0.3]],  # Replace with your embedding vectors
    metadatas=[{"key": "value"}],  # Replace with your metadata
    ids=["your_id"]  # Replace with your custom IDs
)

Make sure to replace placeholders like username, password, your_database, docs, your text data, and others with your actual data. Additionally, ensure you have the pgvector extension installed in your PostgreSQL database ^[1][2][3].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[dosubot[bot]]

To use documents retrieved by similarity_search_with_score as a retriever in a chain, you can wrap the method in a function that adds the scores to the document metadata and then use a @chain decorator to make it a runnable retriever. Here's how you can do it:

from typing import List
from langchain_core.documents import Document
from langchain_core.runnables import chain

@chain
def retriever(query: str) -> List[Document]:
    docs, scores = zip(*vectorstore.similarity_search_with_score(query))
    for doc, score in zip(docs, scores):
        doc.metadata["score"] = score

    return docs

This function can then be used in a chain to retrieve documents with their similarity scores included in the metadata ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[deepak-kumar-98]

@dosu
how to use this in conversation retreival chain
chain = ConversationalRetrievalChain.from_llm(
llm=llm,
retriever=retriever,
return_source_documents=True,
memory=memory,
combine_docs_chain_kwargs={'prompt': prompt},
rephrase_question=True,
)

[dosubot[bot]]

To use the documents retrieved by similarity_search_with_score in a ConversationalRetrievalChain, you can create a custom retriever function that integrates the scores into the document metadata. This custom retriever can then be passed to the ConversationalRetrievalChain.from_llm method. Here's how you can set it up:

1. Create a Custom Retriever Function:

from typing import List
from langchain_core.documents import Document

def custom_retriever(query: str) -> List[Document]:
    # Perform similarity search with scores
    docs_with_scores = vectorstore.similarity_search_with_score(query)
    
    # Add scores to document metadata
    for doc, score in docs_with_scores:
        doc.metadata["score"] = score
    
    # Return the documents
    return [doc for doc, _ in docs_with_scores]

2. Use the Custom Retriever in the Conversational Retrieval Chain:

from langchain.chains import ConversationalRetrievalChain

# Initialize the chain with the custom retriever
chain = ConversationalRetrievalChain.from_llm(
    llm=llm,
    retriever=custom_retriever,
    return_source_documents=True,
    memory=memory,
    combine_docs_chain_kwargs={'prompt': prompt},
    rephrase_question=True,
)

This setup allows the ConversationalRetrievalChain to use the documents retrieved by your custom retriever, with the similarity scores included in the metadata.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[deepak-kumar-98]

@dosu
how can we use distance_strategy, will i store while storing it into db or while fetching?

[dosubot[bot]]

The distance_strategy in the LangChain library is applied during both storing and fetching data. When storing data, the distance strategy is used to normalize the embeddings if the strategy is COSINE. Similarly, when fetching data, the distance strategy is used to normalize the query embedding if the strategy is COSINE. Therefore, you should consider the distance_strategy both while storing and fetching data ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}