existing chunks and embeddings

[NikhielRahulSingh]

Is there a way for me to pass my existing chunks and embeddings to GraphRAG, I have already chunked my documents and embedded them.

[natoverse]

A couple of thoughts here:

Chunks
Yes, we frequently do this when we need to customize the chunks we send (for example, if we want document metadata repeated on each chunk, we will prepend it to the text). The key here is to pre-split your content with a script, but make sure that those chunks (including anything you add) stay below the GRAPHRAG_CHUNK_SIZE limit you set. Upon ingest, your chunks should automatically remain independent.

Embeddings
Bringing your own embeddings is certainly possible, but may require a little more config/scripting. BTW, It sounds like you may want your raw chunk text content embedded, which GraphRAG does not use (we use entity.description for local search only).

1. Use the GRAPHRAG_EMBEDDING_TARGET and GRAPHRAG_EMBEDDING_SKIP params to adjust which fields are embedded. Use GRAPHRAG_EMBEDDING_TARGET=all and set the skip fields from the list at the top of the page here. The only one done by default is entity.description, and you can skip all of them if you'd like.
2. You'll have to script adding your embeddings where required. If you've embedded the text content yourself, add it to the create_final_text_units.parquet in a column called text_embedding and it should line up with what GraphRAG does.

Warning: if you are using a non-OpenAI embedder, the format/length/etc may be different!

[github-actions]

This issue has been closed after being marked as stale for five days. Please reopen if needed.