Problem with icu_segmenter and ICU4X_DATA_DIR

[sffc]

bakeddata-scripts/main.rs has the following line:

        if component == "segmenter" {
            // segmenter uses hardcoded locales internally, so fallback is not necessary.
            driver.clone().with_fallback_mode(FallbackMode::Hybrid)
        } else {
            driver.clone()
        }

However, when customers generate custom compiled data via a manual invocation of icu4x-datagen and then use it via ICU4X_DATA_DIR, they don't get this condition, and end up generating baked segmenter data that uses fallback. This is exposed via the following compile error:

failed to resolve: could not find `locid_transform` in `icu`
could not find `locid_transform` in `icu`
segmenter_dictionary_w_auto_v1.rs.data(2, 5246158): Actual error occurred here
mod.rs(19, 1): consider importing this module: `use icu_locid_transform::fallback;

@robertbastian

[robertbastian]

Yeah I noticed this in Google3 import as well. We should special-case the segmenter keys in datagen to disable fallback. Could do this in a patch release, otherwise in 2.0 we'll use aux keys anyway.

[sffc]

I'm in favor of making a special case for this in datagen in 1.5 as long as it isn't too much work.