@davidsbatista This sounds great! One idea I had for this is some way to indicate that we'd like to utilize something like NLTK to do sentence splitting. So normally I think the list of separator characters would look like ["\n\n", ".", " "] to accomplish splitting by paragrah, then sentence, and then by word. And I was wondering if we could replace "." with something like "nltk" or some other tag to indicate we'd like to use a separate algorithm to handle the splitting.

What do you think?

Also I wanted to ask will the splitting by separators (e.g. ["\n\n", ".", " "]) be handled using a regex splitter? I think supporting regex would be great so we could provide more complicated separators to better handle complex documents and do things like header detection.

that's a good suggestions, I will take it into consideration

@davidsbatista This sounds great! One idea I had for this is some way to indicate that we'd like to utilize something like NLTK to do sentence splitting. So normally I think the list of separator characters would look like ["\n\n", ".", " "] to accomplish splitting by paragrah, then sentence, and then by word. And I was wondering if we could replace "." with something like "nltk" or some other tag to indicate we'd like to use a separate algorithm to handle the splitting.

What do you think?

I would suggest using "sentence" and we use NLTK's sent_tokenize(text), but I now noticed that @vblagoje implemented something more robust.

I think we could use the SentenceSplitter here, but maybe we can also move it out of that file into some utils package or file so that can be reused by any component that wants to implement some splitting/chunking technique.

What do you say?

Also, this NLTKDocumentSplitter seems to be an exact copy of the DocumentSplitter except that it uses NLTK's sentence boundary detection algorithm. Maybe we could also merge these two in the future?

I would suggest using "sentence" and we use NLTK's sent_tokenize(text), but I now noticed that @vblagoje implemented something more robust.

That sounds good to me!

I think we could use the SentenceSplitter here, but maybe we can also move it out of that file into some utils package or file so that can be reused by any component that wants to implement some splitting/chunking technique.

What do you say?

Yes I also agree. Let's reuse that and move it into utils.

Also, this NLTKDocumentSplitter seems to be an exact copy of the DocumentSplitter except that it uses NLTK's sentence boundary detection algorithm. Maybe we could also merge these two in the future?

This is totally correct! I asked the same question here and it does seem like we would like to merge these two in the future. Sounds like we should open an issue for this.

• I've opened an issue for that one: Unify DocumentSplitter and NLTKDocumentSplitter #8600

• Have a PR to move the SentenceSplitter to it's own file, to be easily reusable refactor: moving SentenceSplitter outside NLTKDocumentSplitter #8599

Hey @davidsbatista~

The idea seems to be exactly how SemChunk works. Just curious if it would be preferable to use the SemChunk or to implement from scratch here? Given that in my experience, semchunk is very well written.

Link: semchunk

Also, can we add support for more chunking methods? Full disclosure: I write a lot of chunking and splitting methods at Chonkie

Thanks! 😊