How to update documents/embeddings stored in a vector database using semantic memory?

[jcreek]

I've had a look through and it's not clear what the intended workflow is for this scenario. Say I've ingested a number of documents, but one has been updated, how would I go about updating the memory?

[dluc]

hi @jcreek you should be able to achieve what you suggest simply by uploading with the same Document ID. If you put multiple files under the same ID you will have to upload all of them, otherwise, those missing will be considered deletions. Typically you might want to have one-file-one-document-ID, to keep things simple.
One idea: if you don't want to persist document IDs, you can generate consistent one "hashing" the original file path+name (or URL), something like: Document ID = SHA256(lowercase(full file path))

[ajay-sainy]

hi @dluc

Looks like there are duplicate documents getting created in Azure AI Search even upon using the same document id. Am I doing anything wrong or there is a bug?

code used to import:

await this.kernelMemory.ImportTextAsync(
    text: contentToIndex,
    documentId: questionId,
    index: this.INDEXNAME,
    tags: new() { { "link", $"{GetUrl(questionId)}" } });

[dluc]

Everything looks good from the screenshot. When a document is uploaded, its content is split into small partitions, also called "chunks", each with an embedding record. It's how vector indexing works.

If your question is not about the chunks, what you observe could be because of the document store used. For instance, if you are using the default document store, which is volatile, the system won't be able to detect that the ID already exists. In this case, try switching to a persistent storage, like SimpleFileStorage on Disk for local tests, or Azure Blobs in the cloud.

[ajay-sainy]

Thanks for the quick response.

After adding .WithAzureBlobsDocumentStorage(azureBlobConfig) while initializing the memoryKernel object, the issue is fixed.