WordBreakIterator::word_type() returns wrong type

[MichalDolata]

I was testing an example from https://docs.rs/icu/latest/icu/segmenter/struct.WordSegmenter.html and I spotted a bug.

If the last character is . preceded directly by word, the word isn't returned in result.

use icu::segmenter::{WordSegmenter, WordType};
use itertools::Itertools;

fn main() {
    let text = "one.";
    let segmenter =
        WordSegmenter::try_new_auto_unstable(&icu_testdata::unstable()).expect("Data exists");
    let words: Vec<&str> = {
        let mut it = segmenter.segment_str(text);
        std::iter::from_fn(move || it.next().map(|i| (i, it.word_type())))
            .tuple_windows()
            .filter(|(_, (_, status))| *status == WordType::Letter)
            .map(|((i, _), (j, _))| &text[i..j])
            .collect()
    };

    println!("{:?}", words);
}

I expect it to return ["one"] but it returns empty vector. But if . is replaced with other punctation mark, for example ? it works as expected.

I'm new to icu4x so I wasn't able to prepare fix for that :/

[sffc]

In one. it appears that we do find a breakpoint between the e and the . -- my initial guess is that the lookahead sees the . and returns a different rule status (since it could be an abbreviation, URL, etc) than ? which is a definite hard stop. @makotokato @aethanyc Can you take a look at what the expected behavior is here?

[aethanyc]

I can reproduce this. I'll look into it.