[Bug] generated result changed when using multiple prompts

[irasin]

branch: main
commit: 8516999
test gpu: Nvidia A10

test code:

from vllm import LLM, SamplingParams

prompts = [
        "Quartz is one of the most common",
        "Building a successful software platform is",
        # "Justice studies is an",
        # "In 1997 Ronald Phillips published",
]

model_path = "huggyllama/llama-7b"
llm = LLM(model=model_path)

sampling_params = SamplingParams(max_tokens=128, temperature=0)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"[Prompt]\n{prompt}\n\n")
    print(f"[Generated text]\n{generated_text}\n\n")

result:

[Prompt]
Quartz is one of the most common


[Generated text]
 most abundant minerals on Earth. It is found in many different colors and is used in many different ways. Quartz is a crystalline mineral that is found in many different colors. It is a very common mineral and is found in many different places. Quartz is a very common mineral and is found in many different places.
Quartz is a mineral that is found in many different colors. It is a very common mineral and is found in many different places. Quartz is a very common mineral and is found in many different places. Quartz is a very common mineral


[Prompt]
Building a successful software platform is


[Generated text]
 a complex task. It requires a deep understanding of the market, the technology, and the business. It requires a team of talented people who can work together to create a product that is both technically sound and commercially viable.
At the same time, it is a task that is often underestimated. Many companies fail to understand the complexity of the task and the time and effort required to build a successful platform.
In this article, we will look at the key factors that contribute to the success of a software platform. We will also look at the key challenges that companies face when building a platform.

but if we uncomment the last two prompt

prompts = [
        "Quartz is one of the most common",
        "Building a successful software platform is",
        "Justice studies is an",
        "In 1997 Ronald Phillips published",
]

result

[Prompt]
Quartz is one of the most common


[Generated text]

minerals on Earth. It is found in
many different colors, including
white, pink, red, purple, blue,
yellow, green, brown, black,
and gray. Quartz is a silicon
dioxide mineral. It is a crystal
that is made up of silicon and
oxygen atoms. Quartz is a
non-metallic mineral. It is
found in many different
forms, including crystals,
amethyst, rose quartz,
citrine, smoky quartz,
and rock


[Prompt]
Building a successful software platform is


[Generated text]
.
Building a successful software platform is.
Building a successful software platform is. Building a successful software platform is. Building a successful software platform is. Building a successful software platform is. Building a successful software platform is. Building a successful software platform is. Building a successful software platform is. Building a successful software platform is. Building a successful software platform is. Building a successful software platform is. Building a successful software platform is. Building a successful software platform is. Building a successful software platform is. Building a successful software platform is. Building a successful software platform is. Building a successful software platform is. Building a successful software


[Prompt]
Justice studies is an


[Generated text]
 that is interdisciplinary in nature. It is a field of study that is concerned with the study of the legal system and the legal process. It is a field of study that is concerned with the study of the legal system and the legal process. It is a field of study that is concerned with the study of the legal system and the legal process. It is a field of study that is concerned with the study of the legal system and the legal process. It is a field of study that is concerned with the study of the legal system and the legal process. It is a field of study that is concerned with the study of the


[Prompt]
In 1997 Ronald Phillips published


[Generated text]
 a book entitled “The Making of a Modern Terrorist” in which he described the process of radicalization of a young man named David Copeland. Copeland was a member of the British National Party (BNP) and was arrested in 1999 for a series of nail bombings in London.
Phillips’ book was a fascinating read and I was surprised to learn that the BNP had a history of violence. I was also surprised to learn that the BNP had a history of violence. I was also surprised to learn that the BNP had a history of violence. I was

Since I didn't set the appropriate sampling parameters, I wasn't expecting to generate very good results.
But I think the first two generated results should not change with the greedy search.

[irasin]

Please correct me if I missed any hyperparameter settings.

[irasin]

I guess maybe the promble is introduced by the batch dimension and attn_bias

When we have multiple prompts, we need to do padding in the worker, and the padding format is pad_on_right
But here in the set_attn_bias func, we only consider the max_prompt_len to generate the attn_bias.
It will add the padded token into the attention computation.

    def set_attn_bias(
        self,
        input_metadata: InputMetadata,
        dtype: torch.dtype,
    ) -> None:
        del dtype  # Unused.
        if input_metadata.attn_bias is not None:
            # Already set by a previous layer.
            return
        prompt_lens = [input_metadata.max_prompt_len
                       ] * input_metadata.num_prompts
        attn_bias = BlockDiagonalCausalMask.from_seqlens(prompt_lens)
        if self.sliding_window is not None:
            attn_bias = attn_bias.make_local_attention(self.sliding_window)
        input_metadata.attn_bias = attn_bias

[ymwangg]

Does #1546 fix your issue?

[irasin]

Does #1546 fix your issue?

LGTM, it works.
Please merge it as soon as possible. @WoosukKwon