Support LSTM segmentation data in DataProvider

[makotokato]

ICU4C and ICU4J use resource bundle format (*.res) that is binary blob. JSON files for LSTM data may be big size, so I would like to use anything binary format for it. Actually ICU4J's LSTM implementation can use *.res, so it is better if we can read it on ICU4X.

[zbraniecki]

@aethanyc can you confirm if this is a blocker or just performance nice-to-have for Segmenter?

[sffc]

We are using Postcard in ICU4X as an analog to *.res in ICU4C. You should use the DataProvider infrastructure to serve your data, and let ICU4X take care of the rest for you. For example, if you find yourself reading runtime data directly from *.json files, you're probably doing something wrong.

[sffc]

To be more specific: if you need to store an array of u16s, for example, you should have a data struct such as

#[icu_provider::data_struct]
#[derive(Debug, PartialEq, Clone, serde::Serialize, serde::Deserialize)]
pub struct LstmData<'data> {
    demo: ZeroVec<'data, u16>,
}

ICU4X is able to serialize that with Postcard to an optimized binary data file. You need to implement a "source" data provider to enable that piece of ICU4X's infrastructure to run.

[aethanyc]

lstm.rs currently consumes the Thai training model using json, so this definitely blocking segmenter from being production ready.

I feel this issue is more like unicode-org/lstm_word_segmentation#16. That is, for the existing Thai and Burmese training data, do we have a binary data format that can be easily plugin into icu4x?

[sffc]

Example of making a new transformer: #885

[aethanyc]

This is maybe related to #1426.