Investigate how textwrap works for East Asian languages #80
Comments
mgeisler
added a commit
that referenced
this issue
Apr 8, 2021
This adds a new optional dependency on the unicode-linebreak crate, which implements the line breaking algorithm from [Unicode Standard Annex #14](https://www.unicode.org/reports/tr14/). The new dependency is enabled by default since these line breaks are more correct than what you get by splitting on whitespace. This should help address #220 and #80, though I’m no expert on non-Western languages. More feedback from the community would be needed here.
mgeisler
added a commit
that referenced
this issue
Apr 8, 2021
This adds a new optional dependency on the unicode-linebreak crate, which implements the line breaking algorithm from [Unicode Standard Annex #14](https://www.unicode.org/reports/tr14/). The new dependency is enabled by default since these line breaks are more correct than what you get by splitting on whitespace. This should help address #220 and #80, though I’m no expert on non-Western languages. More feedback from the community would be needed here.
mgeisler
added a commit
that referenced
this issue
Apr 14, 2021
This adds a new optional dependency on the unicode-linebreak crate, which implements the line breaking algorithm from [Unicode Standard Annex #14](https://www.unicode.org/reports/tr14/). The new dependency is enabled by default since these line breaks are more correct than what you get by splitting on whitespace. This should help address #220 and #80, though I’m no expert on non-Western languages. More feedback from the community would be needed here.
mgeisler
added a commit
that referenced
this issue
Apr 14, 2021
This adds a new optional dependency on the unicode-linebreak crate, which implements the line breaking algorithm from [Unicode Standard Annex #14](https://www.unicode.org/reports/tr14/). The new dependency is enabled by default since these line breaks are more correct than what you get by splitting on whitespace. This should help address #220 and #80, though I’m no expert on non-Western languages. More feedback from the community would be needed here.
mgeisler
added a commit
that referenced
this issue
Apr 14, 2021
This adds a new optional dependency on the unicode-linebreak crate, which implements the line breaking algorithm from [Unicode Standard Annex #14](https://www.unicode.org/reports/tr14/). The new dependency is enabled by default since these line breaks are more correct than what you get by splitting on whitespace. This should help address #220 and #80, though I’m no expert on non-Western languages. More feedback from the community would be needed here.
mgeisler
added a commit
that referenced
this issue
May 2, 2021
This adds a new optional dependency on the unicode-linebreak crate, which implements the line breaking algorithm from [Unicode Standard Annex #14](https://www.unicode.org/reports/tr14/). We can use this to find words in non-ASCII text. The new dependency is enabled by default since these line breaks are more correct than what you get by splitting on ASCII space. This should help address #220 and #80, though I’m no expert on non-Western languages. More feedback from the community would be needed here.
mgeisler
added a commit
that referenced
this issue
May 2, 2021
This adds a new optional dependency on the unicode-linebreak crate, which implements the line breaking algorithm from [Unicode Standard Annex #14](https://www.unicode.org/reports/tr14/). We can use this to find words in non-ASCII text. The new dependency is enabled by default since these line breaks are more correct than what you get by splitting on ASCII space. This should help address #220 and #80, though I’m no expert on non-Western languages. More feedback from the community would be needed here.
mgeisler
added a commit
that referenced
this issue
May 2, 2021
This adds a new optional dependency on the unicode-linebreak crate, which implements the line breaking algorithm from [Unicode Standard Annex #14](https://www.unicode.org/reports/tr14/). We can use this to find words in non-ASCII text. The new dependency is enabled by default since these line breaks are more correct than what you get by splitting on ASCII space. This should help address #220 and #80, though I’m no expert on non-Western languages. More feedback from the community would be needed here.
|
I'll close the issue now: the new The assert_eq!(UnicodeBreakProperties.find_words("CJK: 你好").collect::<Vec<_>>(),
vec![Word::from("CJK: "),
Word::from("你"),
Word::from("好")]);These words are then wrapped by measuring their width and putting them into lines as usual. If someone with experience in these languages finds problems, then I would love to hear about them since I'm not an expert at all in these things. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I wonder if it would be possible to tweak the wrapping so as to follow some of the rules mentioned here:
https://en.wikipedia.org/wiki/Line_breaking_rules_in_East_Asian_languages
The text was updated successfully, but these errors were encountered: