tokenizers 0.13.2 does not compile when default features are turned off

[jneuff]

I cannot compile version 0.13.2 with default features turned off.
Steps to reproduce:

$ rustup show
Default host: x86_64-unknown-linux-gnu
rustup home:  /home/julius/.rustup

stable-x86_64-unknown-linux-gnu (default)
rustc 1.65.0 (897e37553 2022-11-02)
$ cargo new try-tokenizers
$ cd try-tokenizers
$ cargo add tokenizers --no-default-features
$ cargo build
...
   Compiling tokenizers v0.13.2
error[E0432]: unresolved import `onig`
 --> /home/julius/.cargo/registry/src/github.com-1ecc6299db9ec823/tokenizers-0.13.2/src/utils/onig.rs:3:5
  |
3 | use onig::Regex;
  |     ^^^^ help: a similar path exists: `super::onig`

error[E0433]: failed to resolve: use of undeclared crate or module `onig`
  --> /home/julius/.cargo/registry/src/github.com-1ecc6299db9ec823/tokenizers-0.13.2/src/utils/onig.rs:12:60
   |
12 |     pub fn find_iter<'r, 't>(&'r self, inside: &'t str) -> onig::FindMatches<'r, 't> {
   |                                                            ^^^^ use of undeclared crate or module `onig`

Some errors have detailed explanations: E0432, E0433.
For more information about an error, try `rustc --explain E0432`.
error: could not compile `tokenizers` due to 2 previous errors

[Narsil]

You need the onig feature.
It's made optional because there is the unstable_wasm build which uses fancy_regex as a replacement for onig.

You can try fancy_regex but it is not the standard regex engine so bugs might creep up (hence the unstable_ for the wasm builds.

[jneuff]

That works, thanks!

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[alfredodeza]

I just ran into this problem, not sure if the docs could get an update or an easier/friendlier solution could be in place for folks that run into this and can't figure out what to do like in my case

[ArthurZucker]

Down to add a warning in the doc! Do you want to open a PR @alfredodeza ?