Incrementally decode output tokens

[WoosukKwon]

Fixes #119

This PR reduces the overhead of the tokenizer by incrementally decoding the output tokens at every step. Specifically, the detokenization process is split into two stages:

1. Convert token ids to token strings (i.e., List[int] -> List[str])
2. Concatenate the token strings to the output text (i.e., List[str] -> str)

In the new method introduced by this PR, the first stage is performed incrementally: the new token id is decoded into a string token and is appended to the list of previous string output tokens. Meanwhile, this PR does not change the second stage: the tokenizer will transform the whole output tokens into the output text at every step. IIUC, this stage cannot be performed incrementally.

# opt-13b inference latency (bs 8, input 32, output 128)
# Main
Avg latency: 3.57 seconds
Tokenizer (fast): 0.14 seconds
# This PR
Avg latency: 3.47 seconds
Tokenizer (fast): 0.03 seconds

# llama-13b inference latency (bs 8, input 32, output 128)
# Main
Avg latency: 5.28 seconds
Tokenizer (slow): 1.97 seconds
# This PR
Avg latency: 3.77 seconds
Tokenizer: 0.48 seconds

Note that there are still some overheads from the tokenizer in case of LLaMA.

[WoosukKwon]

@zhuohan123 This PR is ready for review.

[zhuohan123]

Nit:

Suggested change

	if not (hasattr(tokenizer, "added_tokens_encoder") and
	tokenizer.added_tokens_encoder):
	if not getattr(tokenizer, "added_tokens_encoder", None):

[zhuohan123]

In addition, this optimization seems non-necessary? If the tokenizer has an empty added_tokens_encoder, the code below also only calls convert_tokens_to_string once on the final current_sub_text.

[WoosukKwon]

Thanks for the simplification. Changed the code.

[WoosukKwon]

In addition, this optimization seems non-necessary? If the tokenizer has an empty added_tokens_encoder, the code below also only calls convert_tokens_to_string once on the final current_sub_text.

@zhuohan123 Surprisingly, the optimization does reduce the latency from 4.4 s to 3.8 s. The reason is the python loop over output_tokens takes some time even if the loop body is very simple. I've added more comments on this.

[zhuohan123]

LGTM!