Add chunking support to vectorize.table()

[asr2003]

Closes #142
/claim #142

[asr2003]

@ChuckHend Have a look at your free time and kindly approve workflows, if any adjustments needed i am happy to incorporate them

[asr2003]

Waiting for a quick review on this too - @ChuckHend

[asr2003]

@ChuckHend Any changes needed here?

[ChuckHend]

I might need to provide further clarification in the issue. What is intended is that when calling vectorize.table() on an existing table, each row is split into multiple 'chunks' (one new row for each chunk). I think these new rows will need to live on a separate table.

[asr2003]

@ChuckHend Updated the changes with intended approach of chunking vectorie.table by retreiving rows and lives into new table,

Have a review on it and If any changes or modifications required I am ready to incorporate them

[asr2003]

@ChuckHend Have a review on it in your free time

[ChuckHend]

Can you add a test or two that shows how the functionality will work and assert that it functions as expected?

[asr2003]

Sure! I will update the tests

[asr2003]

Just a minor query just does we need automatic chunking once table is called and it sync with chunk_table?

…

On Mon, 4 Nov, 2024, 8:43 pm Adam Hendel, ***@***.***> wrote: Can you add a test or two that shows how the functionality will work and assert that it functions as expected? — Reply to this email directly, view it on GitHub <#162 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BGXZB6GUWQYZTX2ZY75W7ATZ656JHAVCNFSM6AAAAABQEOKG32VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJUHE3TOMBWGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ChuckHend]

Yes, let's leave the integration with vectorize.table() out to make this simpler / easier. Just keep chunk_text() a sql function that we can call directly, and add a couple tests that validate its functionality and shows how to use it.

[asr2003]

Okay!

[asr2003]

@ChuckHend Any reviews on it

[ChuckHend]

Ill have this reviewed by end of week

[asr2003]

I haven't pushed my latest changes here which are in my local :)

[asr2003]

@ChuckHend 👋
I have updated the chunk_text function to use the text_splitter crate instead of the previous manual approach. This change makes the chunking more accurate especially for larger documents and ensures we preserve word boundaries while still enforcing a character limit. The previous logic sometimes caused inconsistencies with expected vs actual outputs. Using text_splitter brings more stability and aligns with the best practices seen in LangChain’s recursive text splitter. I have also updated docs of it's usage as standalone sql function and added couple of tests that validate its functionality and tests passes locally.

test test_chunk_text ... URL: ***localhost:28817/postgres
ok

With this I think it's ready to merge

[ChuckHend]

is this function being used anywhere?

[asr2003]

Nope! It used in previous approach, will remove it