Add content chunkers

[andreibondarev]

No description provided.

[andreibondarev]

@rickychilcott @alchaplinsky Is this something that could be useful here? https://github.com/moekidev/baran

@moekidev

[kawakamimoeki]

I think the Langchainrb's module style is excellent. I actually have a project where I'm using only the chunker without Langchain's LLM clients.

[andreibondarev]

@moekidev i was just asking if we could use your Baran library in this Langchain.rb project.

[kawakamimoeki]

@andreibondarev Sorry. I hope so. I just wanted to say that I think it would be better if this Langchainrb didn't have a chunker internally, regardless of whether Baran is useful or not.

[andreibondarev]

Oh! Why do you say that? Why not have chunkers here, in Langchainrb?

[kawakamimoeki]

@andreibondarev I don't think it's a good idea to make everything in-house, like versions of Python or TypeScript. For example, even for a chunker, commits get dispersed because they overlap with other LLM full stack libraries. I think it's more effective to have the community commit as an independent library, as this can ultimately lead to richer and safer features. So, regardless of whether it's Baran or not, I think it would be better to call on an external chunker library. This is the reason why I chose to release Baran instead of committing to Langchainrb.

[rickychilcott]

@moekidev - this is an interesting take. I would like to design a chunking system in Langchain.rb that will allow for simple chunking, but allow developers to utilize their own chunking systems as needed.

Given the work you've done on Baran, when I build this out, I may just use your libraries under the hood for our standard chunking, for now. This will allow me to focus on the interfaces of the classes and not re-invent the wheel. This will perhaps be most fruitful in allowing us to then be compatible (or at least play nicely) with other external libraries.

I get the idea that Baran has utility outside of Langchain.rb so a separate gem is useful, but I think we want to provide a good development experience, batteries included, without having to find, locate, and stitch together many libraries -- especially for new AI/ML developers.

[kawakamimoeki]

@rickychilcott @andreibondarev That sounds good. If we maintain these perspectives, Langchainrb might be able to achieve a more sustainable design than the other language versions! Should I try adding a class that uses Baran to Langchainrb on my end?

[andreibondarev]

Should I try adding a class that uses Baran to Langchainrb on my end?

@rickychilcott is in middle of it right now I think!

[mael-ha]

Hey there ✌️
Quick feedback on this matter as I was working with this gem actually had the issue were a large .pdf was exploding the allowed context (260k tokens vs 8k allowed).
Then I discovered this issue and the Baran gem (thanks @moekidev for this 🙏). This gem does perfectly the job. Here is my implementation.

The problem when adding data is actually using PDFReader while processing large file, since it means we are processing the whole .pdf in chunks before sending them to pinecone, then we (at least I did) run into a runtime error on pinecone (and probably other vector databases) side since there is a large amount of data to upload and the connection will be established for a while.
In my case it happened while processing Capital from Marx (500+ pages 👀😂), but the runtime error actually happened around page ~25 (with approx. ~8 chunks / page using baran). I'm in Bali and internet is not great, but still this issue would happen.

Another solution to process large files could be:

1. splitting the .pdf in pages (using CombinePDF)
2. store pages as temp file
3. process each of them (create chunks w/ Baran and upload to vector database)
4. then delete each file when processed.

I'll proceed with that on my side, that could prob be a solution for large .docx files as well. WDYT?

[rickychilcott]

I know I took a stab at #40 a while ago but I just haven’t had the time to dedicate to trying again after some conversations and some of the major class changes. I’ve got some other things that I’m working on for this project and for my primary job. But I don’t imagine I’ll be able to attempt again for another week or two.

So if anyone wants to pick this up.. by all means. I’ve held this hostage for too long. Sorry.

[andreibondarev]

I went ahead and added a basic text chunker: #188. I'd like to get some feedback and then we can figure out what the next iteration looks like. For example we want may to automatically pick a chunking strategy based on the original file format.

I'm closing this issue for now.