fixbug: Expand sentence-ending punctuation check in _create_statements method #1523
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This pull request addresses an issue where non-English punctuation (e.g., "。", "!") was not properly handled in the
_create_statements
function, causing sentences in languages like Chinese to be ignored and potentially leading to incorrect faithfulness scores.Changes Made
Updated the condition:
to
This ensures that sentences with non-English punctuation are included in the processing.
Impact
This change allows the system to correctly process sentences in languages like Chinese, preventing empty
sentences_with_index
variables and ensuring accurate faithfulness scores.Let me know if this works or if you need further edits! Thanks!