[Fix] #37: Incorrect indexing when repitition is present in the text #87
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Updated the TokenChunker class to replace the _process_batch method with _create_chunks for improved clarity and functionality. - This change enhances the overall structure of the code and aligns with recent refactoring efforts in the chunking classes.
- Updated the _create_chunk method to include current_index as a parameter for better control over chunk starting index. - Adjusted the logic in the chunking process to utilize the new current_index parameter, enhancing the accuracy of chunk creation. - This refactor improves code clarity and maintains consistency with recent changes in other chunking classes.
- Removed unnecessary space adjustment in position calculation for sentences, as they are already separated by spaces. - Commented out the _prepare_sentences method to streamline the class and focus on the essential functionality. - This change enhances code clarity and prepares for future improvements in sentence processing.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request includes several changes to the
src/chonkie/chunkermodule, focusing on improving the chunking process and refactoring the code for better readability and maintainability. The most important changes include modifications to the_prepare_sentencesmethod, the introduction of the_create_chunksmethod, and updates to the chunking logic in bothtoken.pyandword.py.Changes to chunking process:
src/chonkie/chunker/sentence.py: Removed the_prepare_sentencesmethod and its associated logic, which was responsible for preparing sentences with estimated or accurate token counts. This method is now commented out.src/chonkie/chunker/token.py: Introduced a new_create_chunksmethod that packages texts asChunkobjects and returns the result. This method is used to simplify thechunkmethod. [1] [2]Code refactoring:
src/chonkie/chunker/word.py: Updated the_create_chunkmethod to include acurrent_indexparameter, allowing for more accurate chunk creation by finding the start index from the current index.src/chonkie/chunker/word.py: Modified thechunkmethod to use the updated_create_chunkmethod with thecurrent_indexparameter, ensuring correct chunk overlaps and maintaining the current index throughout the process.These changes collectively improve the accuracy and efficiency of the chunking process while making the codebase easier to maintain.