processor.tokenize support non-lang-syms

[yzmyyff]

We can assign --non-lang-syms in the old io framework, such as text2token.py and format_data. It leaves a special symbol([laugh] or [noise]) as it is.

But in uio we don't support this feature.

Do we have any plan for this? Or how do I achieve the same purpose?

[robin1001]

I didn't go through the details of --non-lang-syms. do you have the bandwidth to support that?

[yzmyyff]

I'd like to but I'm not free recently. Maybe next week or after.