Simple text chunking

[andreibondarev]

Simple text chunking using the baran gem.

Example usage:

irb(main):001:0> Langchain::Loader.new("spec/fixtures/loaders/sample.docx").load.chunks
=> [{:text=>"Lorem ipsum \n\nLorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc ac faucibus odio.", :cursor=>0}, {:text=>"Vestibulum neque massa, scelerisque sit amet ligula eu, congue molestie mi. Praesent ut varius sem. Nullam at porttitor arcu, nec lacinia nisi. Ut ac dolor vitae odio interdum condimentum. Vivamus dapibus sodales ex, vitae malesuada 
... }]

or

irb(main):019:0* text    = Langchain::Loader.load('/Users/andrei/Downloads/agreement.pdf')
irb(main):020:0> chunker = Langchain::Chunker::Text.new(text.value)
irb(main):021:0> chunker.chunks
#=> [{ cursor: , text: }, { cursor: , text: }, ...]

[alchaplinsky]

Nice 👍

[andreibondarev]

@alchaplinsky We should figure out how the output of this chunks method plugs into the add_data or add_texts when indexing data into vector search DBs

[alchaplinsky]

@andreibondarev Do different vector search databases require different chunk size?

[andreibondarev]

@andreibondarev Do different vector search databases require different chunk size?

It's not at the vector search DB level, it's a concern for LLMs.

For example when data is added to Chroma DB (and all other vector search DBs (almost) work the same way) here the text itself is added along with the associated embedding.

If we generate 1 embedding from the whole large text and then that whole text gets retrieved and passed to the LLM to synthesize the answer. The prompt literally looks like this:

Context:
{context} # The whole text here

Question: {question}

Answer:

The whole text might exceed the LLM context window so it needs to be split into smaller chunks.