Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some question about add_window #13

Open
idiotTest opened this issue Apr 8, 2024 · 2 comments
Open

Some question about add_window #13

idiotTest opened this issue Apr 8, 2024 · 2 comments

Comments

@idiotTest
Copy link

Hello.I think i don't know about the function add_window.There's some code that makes me wonder,“idx < len(doc) - window_size”. Actually,I know it's a critical condition,It makes the last window retain size.But I think,in the bottom of the code,there is " item.metadata[f"{window_idx_name}_lower_bound"] = min(window)
item.metadata[f"{window_idx_name}_upper_bound"] = max(window)",
we use the min and mix func to deal the bound.So i guess the condition “idx < len(doc) - window_size” does not seem to make much sense.

@junruxiong
Copy link
Owner

junruxiong commented Apr 15, 2024

The main reason I added this condition is that when the number of elements at the end of doc list is not enough to create a new large chunk index, I will combine those last few elements into the previous large chunk index. You can reference this graph below:

image

@idiotTest
Copy link
Author

ok,that is what i guess.Thanks for your reply,Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants