Integrate late chunking for (potentially) missing context problem.

[muazhari]

I still don't know if the Colpali model is only contextual to each or all documents. If only contextual to each document, then we could integrate late chunking to maximize the ColPali performance. Currently, this late chunking method is easier to implement, more efficient, and more robust to missing context than traditional chunking methods, as ColPali still keeps 1-page chunking. These might further remove ColPali from the need for pre-processing pipelines.

Details:

• https://jina.ai/news/what-late-chunking-really-is-and-what-its-not-part-ii
• https://arxiv.org/pdf/2409.04701

[ManuelFay]

Very good idea, it's already in the roadmap along with exploring CDE ! Thanks for the suggestion !

[jvlinsta]

Would this mean that you would do "patching" on the embedding space, rather than pixels?
Is there currently some hyperparameter that restricts the chunking to a single page?