Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Unicode 15.0 sentence segmentation #4213

Merged
merged 32 commits into from
Oct 26, 2023

Conversation

eggrobin
Copy link
Member

@eggrobin eggrobin commented Oct 24, 2023

Fix #4038.

Seems to work now; tested with 1 000 000 test cases (I’m not going to check that in) and the last failure I encountered so far was #‌79. I should try an overnight test though.

Outline of the monkey testing process:
In ICU, at https://github.com/eggrobin/icu/tree/export-monkeys-15.0:

# Note that you get one tenth of the loop count for sentence segmentation.
.\icu4c\source\test\intltest\x64\Release\intltest.exe "rbbi/RBBITest/TestMonkey@type=sent seed=1729 loop=100000 scalars_only export=meow"

In ICU4X:

cd components\segmenter
git checkout la-vache/main .\tests\testdata\SentenceBreakExtraTest.txt
Add-Content .\tests\testdata\SentenceBreakExtraTest.txt (Get-Content ..\..\..\icu\meow_sent.txt)
cargo test sentence_break_extra_test

Stare at the weeping tests (you will want to cherry-pick the traces from 21aca87), fix something in the state-transition table,

cd ..\..; cargo make bakeddata components/segmenter; cd components\segmenter

And back to

cargo test sentence_break_extra_test

Repeat until the heat death of the universe.

@eggrobin eggrobin force-pushed the import-monkeys branch 2 times, most recently from 717632c to ae91e72 Compare October 24, 2023 14:01
@eggrobin eggrobin changed the title [Very WIP] Fix Unicode 15.0 sentence segmentation Fix Unicode 15.0 sentence segmentation Oct 24, 2023
Co-authored-by: Robert Bastian <4706271+robertbastian@users.noreply.github.com>
@eggrobin eggrobin marked this pull request as ready for review October 24, 2023 15:47
components/segmenter/Cargo.toml Outdated Show resolved Hide resolved
components/segmenter/Cargo.toml Outdated Show resolved Hide resolved
eggrobin and others added 7 commits October 24, 2023 18:50
Co-authored-by: Robert Bastian <4706271+robertbastian@users.noreply.github.com>
Co-authored-by: Robert Bastian <4706271+robertbastian@users.noreply.github.com>
robertbastian
robertbastian previously approved these changes Oct 25, 2023
Copy link
Member

@robertbastian robertbastian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from what I can tell, which excludes the rules themselves

@eggrobin eggrobin merged commit 09642d9 into unicode-org:main Oct 26, 2023
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sentence segmentation is incorrect
2 participants