Add web binding Tokenizer.tokenToId()

[grf53]

In Tokenizer class of web binding, there is no tokenToId() method while idToToken() is.

There are already the original functions corresponding to tokenToId().

• HFTokenizer.TokenToId()
• SentencePieceTokenizer.TokenToId()

So I simply added function to use them on web.

Not sure logic related to unk_id() that I appended to the SentencePieceTokenizer.TokenToId() is consistent.
So please leave comments if there is any point to modify.

Thanks.

[CharlieFRuan]

The web portion looks good to me. cc @Ubospica for the change on SentencePieceTokenizer.TokenToId()

[grf53]

The web portion looks good to me. cc @Ubospica for the change on SentencePieceTokenizer.TokenToId()

@CharlieFRuan
Thanks for your comment. Alright to merge? or we need more reviews?

[Ubospica]

Hi @grf53, sorry I didn't see this message before. That looks good for me, just one thing to point out.

[Ubospica]

Shall we still align with sentencepiece and still return unk when the token is not found?

[grf53]

Hello @Ubospica. Thanks for your comment!
I am not sure about current circumstances with sentencepiece, but I just wanted to enable tokenToId in web.
As I can see in the following description of Tokenizer.TokenToId(), -1 would be returned when it could not find the corresponding id.

tokenizers-cpp/include/tokenizers_cpp.h

Lines 65 to 68 in 5de6f65

	/*!
	* \brief Convert the given token to its corresponding id if it exists. If not, return -1.
	*/
	virtual int32_t TokenToId(const std::string& token) = 0;

The mentioned function is implementing that virtual method, so I wanted to make it as that.
(For HFTokenizer, it is already covered as I saw.)

As I understood, UNK token from sentencepiece tokenizer could be sth other than -1 due to the configuration. So I added explicit check with sentence_piece_ .unk_id(). But as I mentioned in the first comment, I am not sure this logic is really consistent considering practical uses, so you can suggest me to modify or revert that part to align it.

[Ubospica]

Hello @grf53, thank you for your explanation!

I think we should still align with Huggingface tokenizers, which means returning unk_token instead of -1 when the token does not exist. Additionally, please add tests to ensure that unk_token is returned for non-existent tokens.

For the current implementation of HFTokenizer, we will later modify it to return unk_token instead of -1.

Thanks again for your contribution!