[stdlib] Fix implementation of Unicode text segmentation for word boundaries #83314
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@swift-ci test |
|
Linux: D'oh |
|
@swift-ci test |
Azoy
approved these changes
Jul 31, 2025
|
@swift-ci smoke test macOS platform |
|
@swift-ci test macOS platform |
|
Hm |
|
@swift-ci test |
…ndaries Carefully overhaul our word breaking implementation to follow the recommendations of Unicode Annex swiftlang#29. Start exposing the core primitives (as well as `String`-level interfaces), so that folks can prototype proper API for these concepts. - Fix `_wordIndex(after:)` to always advance forward. It now requires its input index to be on a word boundary. Remove the `@_spi` attribute, exposing it as a (hidden, but) public entry point. - The old SPIs `_wordIndex(before:)` and `_nearestWordIndex(atOrBelow:)` were irredemably broken; follow the Unicode recommendation for implementing random-access text segmentation and replace them both with a new public `_wordIndex(somewhereAtOrBefore:)` entry pont. - Expose handcrafted low-level state machines for detecting word boundaries (_WordRecognizer`, `_RandomAccessWordRecognizer`), following the design of `_CharacterRecognizer`. - Add tests to reliably validate that the two state machine flavors always produce consistent results. rdar://155482680
lorentey
force-pushed
the
pushing-word-boundaries
branch
from
ab558c2 to
bb4e6ea
Compare
|
@swift-ci test |
lorentey
force-pushed
the
pushing-word-boundaries
branch
from
bb4e6ea to
847df72
Compare
|
@swift-ci test |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Carefully overhaul our word breaking implementation to follow the recommendations of Unicode Annex #29. Start exposing the core primitives (as well as
String-level interfaces), so that folks can prototype proper API for these concepts._wordIndex(after:)to always advance forward. It now requires its input index to be on a word boundary. Remove the@_spiattribute, exposing it as a (hidden, but) public entry point._wordIndex(before:)and_nearestWordIndex(atOrBelow:)were irredeemably broken; follow the Unicode recommendation for implementing random-access text segmentation and replace them both with a new public_wordIndex(somewhereAtOrBefore:)entry pont._WordRecognizer,_RandomAccessWordRecognizer), following the design of_CharacterRecognizer.rdar://155482680