support do_sample parameter

[irexyc]

Motivation

• merge EngineGenerationConfig / GenerationConfig
• align gen_config logic with transformers
• prevent modification of gen_config.random_seed

# transformers batch generate
from transformers import LlamaTokenizer, AutoModelForCausalLM
import torch

tokenizer = LlamaTokenizer.from_pretrained("/mnt/140/llama2/huggingface/llama-2-7b-chat/")
model = AutoModelForCausalLM.from_pretrained("/mnt/140/llama2/huggingface/llama-2-7b-chat/", torch_dtype=torch.float16, device_map="auto")

tokenizer.padding_side = "left"

# Define PAD Token = EOS Token
tokenizer.pad_token = tokenizer.eos_token
model.config.pad_token_id = model.config.eos_token_id

# use different length sentences to test batching
sentences = [
         "Hello, my dog is a little",
         "Hello, my dog is a little",
 ]

inputs = tokenizer(sentences, return_tensors="pt", padding=True).to(model.device)

output_sequences = model.generate(**inputs, do_sample=False, max_new_tokens=20)
out = tokenizer.batch_decode(output_sequences, skip_special_tokens=True)
print(f'{out[0]}\n{out[1]}\n')

output_sequences = model.generate(**inputs, do_sample=True, max_new_tokens=20)
out = tokenizer.batch_decode(output_sequences, skip_special_tokens=True)
print(f'{out[0]}\n{out[1]}\n')

[lvhan028]

May conclude the demo results

[lvhan028]

api_server.py should also be updated.
The following is how api_server instantiates GenerationConfig. But now do_sample is added and its default value is False.
So api_server will always request greedy search.

    gen_config = GenerationConfig(
        max_new_tokens=request.max_tokens if request.max_tokens else 512,
        logprobs=request.logprobs,
        top_k=request.top_k,
        top_p=request.top_p,
        temperature=request.temperature,
        repetition_penalty=request.repetition_penalty,
        ignore_eos=request.ignore_eos,
        stop_words=request.stop,
        skip_special_tokens=request.skip_special_tokens,
        random_seed=random_seed)

[AllentDan]

shall we make it a request option?

[lvhan028]

I think it's unnecessary since users can still use top_k=1 or temperature=0. to invoke greedy search

[lvhan028]

stop_words could be a list of list, which can be use set

a = [[123], [456]]
b = set(a) # TypeError: unhashable type: 'list'

[irexyc]

Why stop_words could be a list of list?

[lvhan028]

I tested the following case:

gen_config = GenerationConfig(
    stop_words=chat_template.stop_words,
    stop_token_ids=[[92542], [92540]]  # list of list
)

[lvhan028]

May assert self.stop_words_ids and self.bad_words_ids None for now.
Otherwise, it probably brings in side effects if users input some unexpected token_id.