You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hm. Do you have other examples? Is it always with the double underscore?
Please send us a pull request with your fix if you can. To test a fix to the tokenizer, what we do is run the old and new version on 100,000 tweets, then look at the differences if any.
for instance, pls RT__Tell will be parsed to pls R T__T ell
I have an ad-hoc fix for now. It seems OK to me.
The text was updated successfully, but these errors were encountered: